Fredo had to go. I think that Michael was willing to let his brother live in exile, until the scene in the boathouse. Fredo revealed two things in that scene:

1. The depth of his resentment and jealousy.

2. Knowing that Questadt was Roth's man revealed a far greater knowledge of Roth's empire than a casual "Johnny said there would be something in it for me."

He envied Michael his brains, his position and his family (he tells Michael how he wishes Deanna was more like Kay). He is even convinced that his parents loved Michael more (Fredo was passed over by his father and his mother told him he was left by gypsies). The depths of Fredo's anger and envy, in addition to his past betrayal, meant that he would always be a threat to Michael and to the family. He had to be killed.