We don't see much of his sweetness in the trilogy. But there are some hints: When Sonny, Clemenza, Tom and Tessio go downstairs after having talked to Vito, Fredo stays with him. I think he really cared for his father. Vito turned away in disgust and (according to the novel) later put him on his "shitlist". Just because Fredo didn't fit in his expectations. Vito was apparently much more generous towards Johnny Fontane.
When Michael came to Las Vegas, Fredo arranged a party. Michael in turn didn't care about it. He just came for business. Obviously, he didn't even feel like talking to his brother whom he hadn't seen for a long time.