The greatest boss in NY was probably The Chin. He came at a time when the FBI was really going for the jugular of LCN and he still managed to run his ship relatively smoothly. Very few rats and the cases against them were usually weak. The Chin, due to his cunning ways and clever thinking, was able to stay on the street for sixteen years as boss and he did it at a time when it was virtually impossible to do so.

Outside NY, I would say it's a tie between Accardo and Trafficante. Relatively no jail time and they were able to remain powerful till the end.