Joe Colombo got slapped across his face. In public, in front or others. He didn't do anything to retaliate. He just sat there and took it.

Maybe it had something to do with who slapped him - Carlo Gambino.

(True story - it happened in a bar in Brooklyn a few months before Colombo was shot).