Now that you mention it “there is no hint, either in the films or the novel, that either Vito or Michael philandered” explains why Michael was not pleased with Fredo's welcome party
Tom and Neri also seemed uncomfortable perhaps realising what was coming....

In fairness to Michael, it was a shock to the system!

Michael also seemed taken aback, unaware of Fredo's philandering