Woltz meant that he would look ridiculous to his peers, for allowing this girl that he was taking under his wing and building up to be a star (as well as keeping for his own personal stable, if you catch my drift...) to leave him for Johnny Fontaine. He would be ridiculed by the others because he's a bigshot producer, and the girl chose a good-looking actor/singer over him, and he did nothing to retaliate despite being able to ruin Johnny's career. He's being a petty asshole, because he's entitled and that's what entitled people do.

If he didn't act, he would have been laughed at with comments such as "Hey Woltz, I hear your star girlfriend ditched you for that guinea, and you rewarded him by giving him the lead in your movie!" and the like would make him lose face with the other bigshot producers and movie stars.

Also, yes he does look stupid by not giving Johnny the part even though he knows he's perfect for it. BUT in that world, he would rather lose revenue from an amazing movie to satisfy personal ego clashes and vendettas. So him NOT giving Johnny the part is seen as a better choice than looking like you can walk all over him without retaliation.

"Hey Woltz, why didn't you have Johnny Fontaine star in that war picture you released? The dames would have ate it up." and Woltz's reply would have been something like "That smooth-talking guinea insulted me. You remember that fine piece of ass I acquired? The one I was makin' into a big star? That sonofabitch Fontaine stole her right out from under my nose like it was nothin'. I couldn't give him the part after that. I hate that sonofabitch, even if it would have gotten more money for the picture, people need to know they can't walk over Jack Woltz."