Richards doesn't have to be racist to say what he said. Standup is all he has left and he's being bothered over and over and trying to dodge it with comedy isn't working while at the same time he's allegedly called a cracker by the guy beforehand. He simply burst and wanted to piss the guy off where it hurts, which would be through his race. Is that racist? No...
I disagree. Richards could just as well have called the heckler an ignorant asshole but he instead chose to attack the heckler on the basis of his skin color. Thats a racist remark.
(Don't get me wrong - the heckler started it by calling Richards a racial epithet and he's no better than Richards, BUT Richards has to own up to his reply to the heckler).