Well...Fredo had been sent out to Nevada not only for his own safety and recovery but to learn the casino business, protect the Family's interest in same and establish a beachhead for further expansion.

Instead he spent too much of his time having threeways with cocktail waitresses and either didn't know or care that Moe Green was evidently skimming. And he also let Moe Green not only verbally but physically humiliate him.

So this was not only bad for the Family's business interests and status but per novel the Don really didn't respect men who put sex before business or engaged in "oddball" sexual escapades. He chided Sonny for letting an adulterous relationship interfere with his judgment.

So Fredo messed up an assignment and managed in his father's eyes to look unmanly while doing so. So by the rules they operated under, Vito had every right to be upset.