Welcome, Just Watching! Thoughtful question!
Yes, it was technically "incest," according to law: Vincent and Mary were blood relation first cousins. The only explanation I can think of was that perhaps Puzo was thinking of Paul Castellano:
Castellano married his childhood sweetheart, Nina Manno, who was Carlo Gambino's sister in law. What's more, Gambino was married to Castellano's sister. And Gambino had married his own first cousin. These complicated ties cemented Castellano's rise in the Gambino Family, much as European royalty intermarried to cement alliances throughout history. In GFIII, it underscores Vincent's ambition to rise in the Corleone Family, and to succeed Michael.
Vincent was a very manipulative fellow, a side of his personality that doesn't get much attention.