Actually I'd guess none of the Scarfo guys feel this way about him at all.

He was always just Chuckie's son and they all went away when Joey was in his early 20's, when he was just a kid to them.

Phil said he would have killed him himself if he wasn't Chuckie's son so that gives you an idea what they thought of him and how he behaved.