I think that's part of it. That scene where Michael asks his mother about losing his family - maybe he was thinking about how he'd been covering for Fredo's lack of true talent for quite sometime. Obviously Michael had given his brother a regime or at least some men to oversee. Fredo was running a brothel, something that probably wouldn't have happened under Don Vito but might have been a concession to Fredo's limited talents. {"He's got a good heart, but he's stupid.") Maybe Michael realized that Fredo wasn't quite as stupid as he'd thought, and in fact knew just enough to present a greater danger than any outside force. It was an impossible position for someone like Michael who had to protect not just the business but his family.