Sonny would not have had Fredo killed, and Fredo never would have betrayed Santino. The relationship between the two was undoubtedly different than the relationship between Fredo and Michael.
But here's what I think: Had Fredo been contrite and said to Michael, "I'm sorry. I didn't know it was going to be a hit." and left it at that, perhaps that could have been the end of it.
But Fredo kept at it. He kept expressing his jealousy and rage at being "Stepped over." AND -- and I think this is crucial -- he KNEW the senator at the hearings was in Roth's pocket. HOW DID HE KNOW???????
This would indicate to Michael that Fredo -- even during that whole confrontation -- was lying, knew more than he chose to tell, and was into Roth DEEPLY. He could no longer be trusted. And in that type of a world, you're either with us or against us. No in between.