Paulie was basically Clemenza’s top soldier and protege(before Pentangeli was created). When Paulie become traitor Clemenza took great offense and was more than happy to kill him when Sonny orders it. Yes He chose Rocco to replace Paulie but obviously Rocco eventually rose to become a high ranking member in the family