Michael had several options: banishment to some very remote place, imprisonment within the compound, or (my personal favorite) institutionalization.

The head of the Family was (as Michael told Kay) "responsible for other people." As Vito's son and Michael's brother, Fredo was one of the people Michael was responsible for. For Michael to place Fredo in a position where Fredo could be manipulated was Michael's failure, and Michael should have taken responsibility for it.