Lucky lost a lot of friends/allies in the fifties, culminating in Costello, whose power had already been rapidly drained by his poor showing in the Kefauver hearings and his subsequent troubles with the IRS, not to mention Genovese breathing down his back for control of the family. Once Adonis was deported and Moretti and Anastasia were killed, Costello and Luciano lost strong allies and a lot of power. Lansky also saw that the writing was on the wall but was smart enough not to get too involved with the Italians so he basically took a backseat for the rest of his criminal career, which also hurt Luciano as he was probably his biggest ally and he still had pull within Vegas and the Jews like Moe Dalitz, Doc Stacher, etc., who had shares in the casinos and were skimming off the top too.