Two things:
First, Geary clearly meant Michael's crime family in both instances when he referred to Michael's "family" in that scene. That Michael chose to interpret it as his biological family is part of the "hypocrisy" that both he and Geary were part of.

Second, Geary said "I want your answer and the money by noon tomorrow," so Michael really didn't have room to simply shrug it off. I'm also taking into account directorial license--his last remark made a good scene-closer.