I imagine that Tom was emotionally torn about leaving. On the one hand he probably felt immense loyalty to the family and to Michael as Vito's progeny. But, he was probably disgusted and disheartened by how Michael had laid waste to Vito's legacy of nobility and honor. Afterall, he did tell Frankie that the Corleone family was like the Roman Empire -once!