Huh. What version Mac/OSX are you using? I've had a Yubikey Nano plugged into various iterations of a MacBook Pro for years and haven't noticed this. The worst battery killer for me is Firefox.
I thought I was being crazy and that maybe something was brushing it, so I let it sit in a moat on my desk immediately after booting. Full battery discharge in sleep in under 30 hours.
Without the yubikey, a full weekend sitting on my desk from a full charge leaves a reported 100% of battery.
