Luca made Vito nervous for several reasons:
First, the novel describes Luca as "...a natural force...like lightning..." He was so fierce, so violent, so un-subject to the usual constraints of behavior, that even Vito, who commanded his loyalty, could not be totally certain that Luca wouldn't do something crazy or at least embarrassing.
Second, Vito wanted Connie's wedding to be a respectable affair. Luca's fearsome appearance (and ever-present violence, which Puzo describes in the novel as "...a violet fury...", could detract from the dignity of the affair. Recall that Kay asks Michael, "Who's that scary-looking guy?" If she noticed, so did others.
Finally: Given Luca's fierce appearance (and probable prominence in law enforcement suspicions), Vito probably feared that being seen in close proximity to Luca might have provided fodder for law enforcement observers. Yes, we see that the "goddam FBI" is outside the compound's gates. But Vito, a (generally) careful guy, might have feared that some law enforcement undercover guy might have infiltrated the wedding and noted his private meeting with Luca. Far fetched, but in Mob life, nothing is totally far fetched.