When Genovese came back to America in 1945, I think Costello was at the height of his power, and could not fathom that anyone, even Genovese would make a move on him. Costello wasn't by nature a violent man so even though he probably suspected Genovese aspired to be Boss once again, there was no reason to ignite a potentially volatile situation by whacking Genovese, who was popular among certain elements of the family in his own right.

Although he was a white collar guy, Costello still at that point had the support of Willie Moretti, his cousin, who had an army of goons at his disposal. Combined with Costello's white collar criminal base plus his political and law enforcement contacts, Genovese knew that any move on Costello would have been suicidal. Genovese wisely bided his time and struck when Costello was at his weakest.

I suppose you could view it as a mistake on Costello's part to not kill Genovese when he still had the muscle behind him.