The character of Clemenza was so specific that I believe there must have been specific inspirations.
For example, the rug theft. Or the mixture of a good-humored person with a cold blooded killer.

The closest I can come is Frank Labruzzo, a ranking member of Joe Bonanno's family. He was loyal to Joe during the "Bananna War," and was very close to son Salvatore (Bill), sharing his on-the-lam apartments and bodyguarding him. But, Labruzzo also had worked in legitimate fields, and even attended college for a while. He was also brother to Joe's wife Fay, and thus Bill's uncle.

Interestingly, Richard Castellano played Labruzzo in "Honor Thy Father," a very good made-for-TV film starring Joseph Bologna as Bill, and the one and only Raf Vallone as Joe.