It's true that Michael, in his generation, might not have expected betrayal by his own brother. But if he had a weakness in this situation, it was that he did not entertain the possibility that the simplicity and weakness that led to Fredo's being passed over might become a source of resentment that would lead to betrayal.

About the closest he came was after the Tahoe shooting when he told Tom, "Fredo? He's got a good heart. But he's weak and stupid, and this is life and death." I took that to mean that while Michael didn't suspect Fredo of outright betrayal at that point, he allowed for the possibility that Fredo might have endangered the family inadvertently, and/or that he didn't have the stuff to run the family while Michael was away. To borrow a phrase from Ian Fleming: he smelled a mouse when he should have smelled a rat.

Stepping back two steps: Michael's real weakness that his obsession with becoming "legitimate" by acquiring Roth's Havana empire blinded him to Roth's treachery--which included recruiting his own brother for betrayal.