I think Vito's remark to Tom is both a father tweaking his son that "Don't think because I'm older that I'm forgetful or didn't do my own investigation" and a not so subtle hint from a boss to a subordinate that his answer to Solozzo would be no.

Vito had contempt for men who allowed their passions to rule them (buyers of sex and/or adulterers, junkies) and even more disdain for people who made money from them. So, especially since his position atop the underworld seemed secure it wasn't that difficult a decision for him to make.

If it were 20 years prior? More difficult to say. He did protect and take into his service a man who killed women and killed his own child. So Vito's "scruples" aren't as important in every circumstance. But no I think the character honestly didn't want to get involved in drug-peddling for moral and practical reasons. He didn't have as much to gain as Solozzo did.