While it's a good idea to move ssh to a different port, if only to stop the annoying doorknob rattling in your log files, you should not pick an ssh port over 1024. Ports over 1024 are unprivileged ports, and if someone could somehow convince your ssh server to die, they could start up a counterfeit one and collect your password. A port under 1024 requires root access to be able to open it.
Of course, this only applies to linux systems where you don't trust local unprivileged users. Or software that you are running as an unprivileged user. So every system.
> they could start up a counterfeit one and collect your password.
you missed the part where I disable password logins on all of my boxes :-) The important point was that the system was already secure enough due to the key requirement, and moving the port was indeed just to stop the "doorknob rattling". If I suddenly find that a box I control is asking me for a password, I'm not going to just type my social security number in and hope for the best.
One could argue that using a port < 1024 makes it easier for the scanners to find, but frankly anything other than 22 (or a frequently scanned port) would work well enough.
Of course, this only applies to linux systems where you don't trust local unprivileged users. Or software that you are running as an unprivileged user. So every system.