You gotta remember that Woltz was a powerful mogul in Hollywood who was used to having his way. When he is seemingly calm at the point Tom is forst talking to him, note how he curtly says "Woltz is listening," when he isnt listening at all, he is looking at a clipboard and signing a piece of paper. When Hagen finishes, Woltz gibes the flack holding the clipboard a smirk, as if to say "get a load of this asshole," and then he raises his voice to Tom.

This kind of behavior probably was a great way to control all those Hollywood finocchios, so Woltz figured he could use it on the Corleones....especially the bit about how he was no bandleader. In a deleted Saga scene Tom discusses with Vito various options, including blackmailing him about the young child star he is sleeping with (note how Vito says under his breath that this is an infamnia) and Vito asks whether this guy is really strong...like a Sicilian or is just bluffing. Tom says he isnt that strong and then he sends for Luca.