I think the second remark might mean that Vito thinks there must be some fundamental flaw in anyone who worked his way up through prostitution.

"All of these men were good listeners; patient men."