I don't think they showed them leave but remember Rick kind of got mad and said they couldn't stay? I THINK they just left cause I don't remember a specific scene showing them leave either. I'll bet they end up catching on to the governor and helping Rick in the end.
TIS
"Mankind must put an end to war before war puts an end to mankind. War will exist until that distant day when the conscientious objector enjoys the same reputation and prestige that the warrior does today." JFK
I'll bet they end up catching on to the governor and helping Rick in the end. 
