He was even more wretched to Tom in the penultimate boathouse scene with Rocco and Neri.

But, he was being logical and truthful after the machine gun attack. Tom really was the only person he could completely trust at that point because, as he said, Fredo wasn't up to running the family in his absence, and he couldn't rule out Rocco and/or Neri at that point. He was also at his most manipulative with his "You're my brother, Tom..." Even more manipulative when he visited Pentangeli and convinced him to stick his head into the lion's mouth to meet with the Rosatos. What a bastard!