I don't think he would have killed him, but he would have basically disowned him. Personally, I think that the reason Michael ultimately decided to have Fredo killed wasn't just because he "took sides against the Family" but because he lied about it, willingly, and then went on his "I was passed over" rant. That rant is what sealed his fate because it showed Michael that Fredo would never be happy just being "the Don's brother"; he would always resent the fact that he wasn't made the Don, whether or not that was "how Pop wanted it". That, IMO, is the real reason Michael had him killed: he could never trust that Fredo wouldn't seize any opportunity or sign of weakness on his part to take another stab at control in the future. If he had been left alive, in the GFIII timeline maybe Fredo would have been manipulated by Altobello and Luchesi instead of Zaza, or maybe he would try to do another hit on Michael, or any number of possibilities. He couldn't be trusted even as a brother after that rant happened.

If Fredo had confessed and went back to supporting his brother, I think he would have gotten a pass but been out of the family and probably given one of the "Mickey Mouse nightclubs" to earn a living. I don't think he would have been killed, but he would have been kept on a very short leash and no longer privy to any information about the family business, so if he ever did decide to try another coup he wouldn't have anything of value to offer.

The most that I think that would have happened would be a mock execution or some similar warning to let him know that was the second time he took sides against the family, and the next time there would be no forgiveness.