Re. Vincent "taking over" the family:
One of the subthemes of GFIII was how Vincent and Michael used each other. Vincent cleverly buttered up Connie to get close to Michael. When Michael seemingly took Zasa's side and ordered Vincent to "make his peace," Vincent forced the issue by biting Zasa's ear, thus making Michael choose between family or "ally." But though Michael appeared to condemn Vincent's hot temper and violence, he was using Vincent as a kind of blunt instrument to let others know that he could still summon drastic action, if needed. Then he put Vincent in mortal danger through the ploy with Altobello. I believe that, if Michael had had his way, Vincent would have been little more than an arms-length underboss--a kind of head of security and an enforcer, to be kept increasingly distant as Michael became increasingly "legitimate."
As for Castellano: Though he wanted to be called "Paul," or "Big Paulie," he really didn't have the respect of his men. He violated Mafia "rules" by carrying on with Gloria Olarte, his Columbian maid, under his own roof, with his wife and daughter resident. He squeezed his people unmercifully, and involved himself in petty matters that should have been handled by capos or lower. He seldom left his Staten Island mansion to meet with his people, demanding that they come to him (and only a few were permitted this "honor"). He also disrespected his esteemed underboss, Neil Dellacroce, by failing to come to his wake or funeral. The last straws: after Dellacroce died, he dissolved his crews (including Gotti's) and named his driver, Tommy Bilotti, a stammering, bat-wielding thug, as his heir-apparent. Castellano's RICO indictment caught him making denigrating remarks about the other NY Mafia Dons. They feared that he'd turn rat rather than spend the rest of his life in jail without his beloved Gloria. That's why they approved Gotti's murder of Castellano.