With Michael talking about his mistress he seemed to get out a lot like Sonny did it seems as it looks like Vito and Michael didn't like any of them fooling around on their wives hence their mentions of it throughout the movie (Vito with Sonny, Michael with Tom). So it seems Tom got around, we just didn't see it.

Crap I forget, did Michael mention it in the novel? I forget. I need to re-read it again since it's been a while.