I finally secured my ssh server after I got hacked twice. The first time was because I had a user named vmware with the password "vmware". The second time was because I had a user named test with the password "test". Yeah I know, not smart. Luckily both those users were not in the wheel group so they were fairly isolated. It looks like the just wanted my box to do port scans and cracking of other machines.
Here are a few things to add to your /etc/ssh/sshd_config file to make it more secure (in addition to the standard defaults):
#Change your port to something other than 22 (security by obscurity).
#Limit which users can log on
#Not sure what this does but it is "turned on for security"
#Enable key authentication
#Disable passwords (force key authentication)
A couple other things I did was to install libpam_cracklib and set a better password for myself. Secondly I installed denyhosts which dynamically adds bad behaving clients to the /etc/hosts/deny list. Beware that it will add svn-over-ssh clients to this list (because svn often makes multiple ssh logins in succession) so you'll need to add the host that you're using svn from to the /etc/hosts.allow list.