Gigante was 21-4 as a boxer. So he definitely wasn't bad.
Your argument doesn't hold water. 1. Pro boxing in the '50s wasn't exactly a clean sport. A young connected fighter's record may not indicate the true worth of the boxer. 2. Who did Gigante fight? Could've been a bunch of old club bums.