In the novel doesn't it state that Vito was uneasy around Luca Brasi, he made him nervous like he was handling dynamite or something to that effect?
Yes.
Vito's insight into Luca (as explained in the novel) is that Luca wants to die. I think Vito's ambivalence toward Luca contains some part of guilt about using him, and some part of fear over Luca's instability. Even while remaining completely loyal, Luca could cause a lot of trouble by accident.
That said, I think we overstate Luca's weirdness on a everyday basis. He plays cards, he consorts with women, he goes with movies. He has cronies. He lives with family. I don't think he was completely isolated in the family.
Finally, while I'm sure that Luca's loyalty was legendary even within Mafia society, I don't think it was such that everyone in all the families would know that he would never turn against the Don, so that everyone would know that his mission among the Tataglia was a ruse.
