TB's analysis seems plausible, but.

Was Fredo really such a good actor in the boathouse?

When he says "I swear to god I didn't know it was going to be a hit. ", he really seems to mean it.
Compare it to Fredo's nervous, bad acting in Cuba.
I think, in the boathouse scene, Fredo is emotionally naked. He speaks frankly about his frustrations as well about his innocence regarding the hit. Just from what I see of Fredo, I believe every word of his in the boathouse scene.