He was testing Fredo's loyalty.

And it didn't really matter whether Fredo failed or not because as we know, the bodyguard had already been dispatched to kill Ola & Roth while Michael and Fredo were attending the New Year's Eve party. What hampered this was Roth having to be taken to the hospital.

And I disagree that Fredo was on the verge of confessing to Michael, although he obviously felt incredible guilt. He knew that what he had done was unforgiveable and he would not dare admit that he had worked with Ola & Roth, thereby enabling the assasination attempt at the Tahoe home.