It has been discussed many times, and there are a great many nuances. I think a part of Michael knew he was going to kill Fredo the minute it was confirmed that he was in league with Ola and Roth. There may have been another part of him that didn't really want to do it, but after the "second betrayal" which Michael exposed in the boathouse, Fredo was a dead man walking.
He didnt want to kill Fredo all the way up until the very end. He only cleared his head right before the actual gunshot. You could see him agonizing a few minutes before he goes to the window to view the shooting.
