I think that move would have cost Geary more than just some temporary loss of votes.
The Corleones could produce lots of witnesses who had seen Geary in the brothels. Maybe even a prostitute who would tell more about the stuff they had done "befower".
Maybe he could pull the Corleones into some mess. But he would drown with them.
Geary was dizzy when Tom talked to him. But when Tom said "All that's left is our friendship", he must have known what was going on, because Tom was there when Geary tried to squeeze Michael, and he must have known the meaning of "friendship" in the Mafia context.