Mike knew a long time before this that Carlo set Sonny up. He may not have been 100% certain of it (and that's why he needed to hear Carlo admit it) but it was Rocco telling Mike that all the Corleone enemies were effectively dealt with in that scene on the church steps.

Welcome to the boards, johnny! (One request - please don't type in all caps).