Fredo was anything but harmless. He showed his true colors when he bitterly told Michael that he resented being passed over and relegated to run mickey mouse night clubs.

Perhaps if Fredo had acknowledged his mistake and sincerely apologized to Michael there could be a case made for clemency, but that didnt happen. In fact Michael had no choice but to take Fredo out. The jealousy was still there, and so was the resentment. It would only have been a matter of time before Fredo betreayed him again. For instance, had Fredo been alive in GF III, who knows what would have happened....perhaps security at the Opera House would be botched and Mary would have ended up dead.