It was Sasha and Rosita that tried to break Eugene and he froze in fear. I don't see Eugene leaving until Negan is dead and even then I think the power might have went to his head.

I still wonder how they bring into the mix whoever he was talking to via c.b. before Negan made him leave Alexandria.