Vito's hands were tied as it relates to Carlo. He took an oath not to do anything to start another war, so if he killed Carlo it would be an indirect attack on the Barzini family.
Vito and Michael meticulously planned what had to happen in the upcoming war (first order of business, build a secret regime under Neri and send Tom to Vegas).

This was always to be Michael's war, so Vito left it to him.
When Michael has had all the heads of the families killed it is then he approaches Carlo and informs him TODAY I SETTLE ALL FAMILY BUSINESS.

I have no doubt Vito in his day would have been ruthles enough to kill Carlo, son in law or not. This is the guy who shot Fanucci in the mouth one time to many, just in case he wasn't dead, and then casually relieved the corpse of a wallet and all his money, in order to take power with the backing of Clemenza and Tessio (who paid him $50 each). This is also the guy who invited that landlord to ask around about him. Once the poor landlord did, he was terrified to so much as sit down in his office. Obviously he had heard some scary stories.