When Tom was walking towards the Boathouse after the brothers hugging at Mama's funeral, Tom looks at Fredo who was showing his Hail Mary fishing techniques to Anthony, Tom must have been thinking, Fredo, you lucky duck! getting a pass or Don't go fishing with Neri!
I think with that shot, Coppola is telling the audience that Fredo is the real subject of the conversation that follows. When Michael says he has to wipe out his enemies, he's talking about Fredo. When he talks about "these things I have to do," he's talking about killing Fredo. Unlike in the first movie, Michael fidgets with his cigarette lighter, so it's not just "business" this time. He maneuvers Tom into pledging loyalty, and Tom obliviously asks, "Now what is it that you want me to do?" The next shot is of Fredo.
