One interesting detail about Fredo's role in the Family is contained in Clemenza's conversation with Sonny after the assassination attempt:

Sonny asks about how often Paulie called off sick and Clemenza says, "I asked Freddy if he wants me to get a different bodyguard and he said no."

That's the only time we see a capo defer to Fredo or Fredo being given authority to make a decision. Fredo's authority to select Vito's driver/bodyguard must have come straight from the Don, with disastrous consequences. (The Don was slippin'.)

Maybe that's why Vito is so dismissive of Fredo after returning to the hospital.