There are two schools of thought, one being the boathouse theory and the other being the "in Havana" theory.

Basically the first one goes along the lines that Michael had extended Fredo the courtesy to have the run of the estate so long as he remained harmless. He tells Tom, after fleeing Cuba to get word to Fredo to tell him he understands that Roth misled him and that he should come back home.

The second theory is that as soon as he found out Fredo was the traitor in the family he decided to kill him sooner or later. the evidence of this is the famous "kiss of death" scene at Batista's New Year's Eve Party.... "I knew it was you Fredo...you broke my heart....you broke my heart."

For some time I believed he withheld the decision to kill Fredo until the boathouse scene where he realizes how deep Fredo's resentment is, and when he learns that Fredo had lied to Tom about what he knew about the Senate Committee. Now I think he decided to do away with Fredo in Havana, but in his typical fashion never shared this with anyone. He did this to keep his enemies closer, and indeed by doing so he was able to get necessary information from Fredo.

After the boathouse scene Michael had no more use for Fredo, so he banished him from the casinos and told him to give him advance notice when he was coming to see his mother. At that point Fredo was really "nothing to [him] now." Then it was just a matter of time for Fredo. Nothing would happen to him while his mother was still alive.