I'll add that Tom raises his voice several times with Sonny and contradicts him without fear. He never does with Michael and seems to be feeling a bit threatened when questioned about the Vegas job offer.
I think by the end of II, everyone is afraid of Michael. Tom sees he's like a barely restrained animal underneath that veneer of calm. Sonny could be violent and temperamental, but he wasn't cruel. I think Tom knew Michael was cruel, in a way neither the Don nor Sonny were, and as such, feared him - Or what he COULD do.
Sonny and Tom are brothers. Fredo is the kid brother they look after and love. Michael is, always was, and still remained the odd man out. Even with Sonny, he seemed awkward and distant and that's his own blood brother. Like, in 1940s scenes before Sonny is killed, Michael just seems a step removed from his brothers emotionally, not even just from the Mafia business.
