Hmm, maybe one "sleeve" of pants is called a pant. Perhaps people used to just wear a pant, either on their right or left leg, and when they decided to attach two together for both legs, they became a pair of pants.

I have no idea how Duck Hunt works - if I give in and look it up, I promise not to tell anyone.