Mock execution is basically dragging him out somewhere, putting an empty gun to his head and firing it, as a warning (this time the chamber was empty, next time there'll be a bullet). Or something similar to that.
I don't think that would have been a good idea. It would only humiliate Fredo even further as a brother and result in even more resentfulness on Fredo's part.
"It was between the brothers Kay -- I had nothing to do with it."
