So, as someone needing to run unattended scripts across a wide variety of hosts, what's the best practice for managing SSH keys? I've been using ssh-agent. Is there a better way?
Make sure that the SSH keys are tightly restricted in the authorized_keys file so they can only run the commands that need to be run unattended. If possible, also restrict them by source IP address. Here's an example authorized_keys file that I have in production for doing unattended backups:
I wouldn't worry unduly about protecting the keys themselves - since they need to be accessible for unattended operation, there's not much you can do to prevent them from being accessible to an intruder.
This is hard, since there is such a large number of them. It would be very painful to maintain this list. For a while I had the keys expire in ssh-agent, but that didn't work because you can't extend the expiration before you run a set of unattended commands. That is, if it's 11.5 hours into a 12 hour expiration, and I need to run commands that will span the expiration, then I'm screwed, since 1) I can't query when the keys will expire, and 2) I can't extend it another 12 hours (from now, not from the expiration).
I feel like ssh-agent is essential to my life but it's badly designed and I have this nagging feeling it's insecure (and articles like this feed this fear).
That sounds like a bit of a quandary. One thing I'm not clear on - if these run unattended, how do you enter the passphrase?
ssh-agent isn't insecure, it's just not magic. If an attacker gets root access to a box, they can examine the system's memory, and the memory of ssh-agent necessarily contains the unencrypted private keys.
Edit: if you're really concerned about private keys being exfiltrated, you can always use a smartcard (the OpenPGP card[1] in a Gemalto USB Shell Token[2] works well with SSH). But if the smartcard is online all the time, then an intruder can always simply use the smartcard to SSH wherever they want, even if they can't actually get the private key itself.
If that's the case, then I would start a dedicated ssh-agent process just for the run, add the identities (without expiration), and later kill the agent when the unattended run is over.
That's fine for that kind of scenario, but often "unattended run" means something spawned by cron, like a daily backup, or a CI system that spawns build jobs on slave machines. There's not a lot of options there.
> I feel like ssh-agent is essential to my life but it's badly designed and I have this nagging feeling it's insecure (and articles like this feed this fear).
If the attacker has root-access, like in this article, they could also recover your decrypted private keys from the memory used in active connections. (SSH and SSL both)
I would hope that ssh zeros the memory used for the private key as soon as it's done with it (which would be after the authentication step in the handshake).
But then it would have to either keep the passphrase in memory (just as bad) or ask for the passphrase each time you reconnect later (defeats the purpose of ssh-agent).
Edit: Nevermind, you're talking about ssh, not ssh-agent..
Not really a solvable problem. You want your machine to have the keys to run scripts but don't want your compromised machine to have the keys to run scripts. The best you can do is restrict what the keys can do or use authentication that doesn't work unattended.
If you're just doing batch jobs, then you could have the script remove keys from ssh-agent when it's done. At a certain point you have to presume the integrity of your machine. Otherwise your password can just be keylogged as you're unlocking key.
