I think the actual soldiers loyal to the Lannisters wen back to Casterly Rock with Tywin's brother since he inherited it after Tywin died. The gold cloaks do what the kings says and since he is a p**sy he didn't send any troops in.

The Dude: And, you know, he's got emotional problems, man.
Walter Sobchak: You mean... beyond pacifism?


Walter Sobchak: This guy f*cking walks. I've never been so sure of anything in my entire life