Tom knew it, too. Right after the Tahoe shooting, Michael says, "Tom, you're my brother." Tom practically breaks down weeping: "I always wanted to be regarded as a brother by you, Mike"--which tells me that he didn't think Michael regarded him as a brother.
Did Sonny? He told him That's easy for you to say, Tom, he's not your father
