I doubt Accardo and Aiuppa were terrified of him. He was a loose cannon and drawing too much attention. Once Giancana was out of the way and they had an excuse, they whacked him.