No high ranking Mafia guy ever carries a gun--too easy for any cop to arrest him for possession of a deadly weapon, just like a common criminal. Fredo was carrying that day because he had to bodyguard Vito in Paulie's absence--and for the dramatic effect when he fumbles the gun as the two guys shoot Vito.
Fredo as Vito's bodyguard -- The Don was really slippin'
Then Fredo must have been licensed to carry firearms
