"I got my own family" was a reference to his brother being in the hands of the Corleones.

I remember John Gotti was once asked if he was the boss of his family and he said "Yeah, I'm the boss of my family, my wife and kids at home."