He went on record with an affidavit about how he "murdered on Michael Corleone's orders." Yes, he retracted the affidavit under pressure of his brother's appearance, but damage had been done. As long as he lived, the government could have tried to use him in some way against Michael.