It's all there in the boathouse scene - all Fredo wanted was something for himself, something not granted as charity by his younger brother. Whatever Roth & co promised would have been some sort of independent money-earner. A chance to make a name for himself. The details don't matter, because it was all a lie, anyway.

And no, I don't believe for a second that Fredo had any idea that his 'treachery' involved Michael's death. He just got hoodwinked by Roth and his people. When Michael says to Tom that Fredo "has a good heart, but he's weak", I've always taken that to imply, "and stupid."

Hyman Roth and Johnny Ola played Fredo for a sucker, which Fredo was. Weak, gullible, vulnerable and jealous.

As far as the attempted hit was concerned, I don't think Fredo had anything to do with that specifically (e.g. letting anyone in, informing anyone where Michael would be). Killing Michael was just a part of Roth's overall plan, which Fredo played a part in at another level.
In Michael's eyes, of course, that was still as bad as if Fredo had led the killers to his bedroom with a torch, a few rounds of ammo and a packed lunch.