This seems like selective logic to me. By this reasoning, divorce should usually be unacceptable too, since most people's vows include "till death do us part" or a variation thereof. It strikes me as odd to basically argue that only one part of the vows is morally binding while the rest of it isn't.

This reminds me of those who oppose gay marriage on the grounds that marriage is about having children, yet have no problem with a 70 year old woman getting married.

Moreover, some marriages seem to be very successful despite the fact that one or both people is known to have affairs. Bill and Hillary Clinton are a good example. One would have to have a lot of chutzpah to argue otherwise.

Not to mention the fact that most people who have affairs don't end up wanting to have them until they're already married.