According to the GF II script, Tom was having an affair with Sandra Corleone, Sonny's widow.
Although there was mention of a mistress in the movie, nothing is mentioned in the book, but it makes for a possible explanation why he consider, "out".