Turnbull brings up a great point; in the novel I think the real reason Woltz was so taken aback was twofold:

1) This "nobody" from New York was able to infiltrate his mansion with its private security as though it was nothing, not only getting to his stable but getting into his bedroom. For someone bragging how they had a "personal relationship" with J. Edgar Hoover and all the power that Woltz thought he had (I believe the novel states that he had dined at the White House), his defenses were nothing. A bigshot was brought low and humiliated by some "obscure importer of Italian Olive Oil"

2) Don Corleone had no issues killing a racehorse worth a lot of money (I can't remember the exact amount, but I think it was several hundred thousand?) simply because Johnny Fontaine didn't get a part in a movie. To Woltz, that was unfathomable because of his lifestyle - who would destroy a magnificent racehorse over something as petty as someone not getting a part in a movie? I think moreso that demonstrated that, beyond the implication that Woltz might be next, that Don Corleone had zero problems with doing everything in his power to ruin Woltz over something so trivial.