Good question, DPN!
You're right--the usual Mafia "rule" is not to whack an honest cop. And, yes, the Bonannos were renegades.
I think they would have whacked him for the most basic reason: survival. They'd have realized that he could send them to prison for decades, but if he disappeared, he wouldn't be able to testify against them. They might have rationalized "breaking the rule" by claiming that he wasn't an "honest" cop because he posed as a wiseguy-wannabe to infiltrate their family.