Several reasons:
First, Roth knew Frankie would be at the party, contentious over Michael's support of teh Rosato brothers against him. Therefore, Frankie would have the "perfect" motivation to kill Michael, and would be an obvious fall-guy. But I believe Michael realized that Roth's plot was too pat, too obvious. Frankie wasn't smart enough, or ambitious enough, to run the Corleone empire. Killing Michael simply to gain freedom to run his NY family wouldn't be worth the trouble that would come crashing down on Frankie afterward. He didn't have enough to gain to take that risk.
Second, I think Michael saw that Roth did have more to gain by killing Michael than Frankie would have. Roth had been partners with Vito, but only in the Prohibition booze business, as Roth reminded Michael several times. There's no evidence that Vito had any part of Roth's Western or Havana gambling empire. But Michael had been horning in on Roth's interests for years. The first thing we saw him do after he returned from Sicily was to try to force Moe Green--Roth's best friend--out of his Vegas hotel; and when he resisted, Michael had him killed. By the time of Anthony's party, Michael had moved his operation to Nevada, owned or controlled three Nevada hotel-casinos, and was pushing Klingman out of a fourth, partly owned by Roth. And he was avid for Roth's Cuban interests. Roth saw that there wasn't room for both of them in the gambling business, which is why he strung Michael along, waiting for an opportunity to have him killed. I think Michael finally realized that after the bullets flew.
Michael probably was 99 percent certain about Roth. But, if he had a lingering doubt, it was erased when he confronted Frankie at his home in NY. When he told Frankie he wanted him to "help me take my revenge," Frankie replied, "Michael..anything... That's when Michael made his big demand, "Settle these problems with the Rosato brothers." That was the ultimate test of loyalty. When Frankie agreed--in effect, putting himself at great risk--Michael was certain that it was Roth all along.