Primo, Fredo flat-out says, in that famous boathouse scene, "I'm your older brother, Mike...and I was passed over!" When Michael replies, "That's the way Pop wanted it," Fredo practically screams, "It ain't the way I wanted it!!" That tells me nothing short of the Donship would have satisfied Fredo. It also tells me Fredo had a strong motivation for wanting his brother dead.
Why didn't Michael know about Fredo's resentment? Because he was too self-centered to notice--or to care. As far as he was concerned, Fredo was weak and stupid, but apparently not dangerous. I can understand why a brother wouldn't suspect another brother of treachery, but Michael simply shut his mind to that possibility. His mistake.