I always took the "look under his fingernails" to mean that there was a big chunk missing from that meeting. I mean, you were there, right? You could tell Sollozzo was hiding something. Vito needed to know what that something was.
Sonny was paying attention to what Sollozzo was saying.
Vito was paying attention to what Sollozzo wasn't saying.