[snakes1100]

nano /etc/hosts.allow

add this in the file and save the file (add a backup IP so you dont have to call your host):
sshd:xx.xx.xx.xx, xx.xx.xx.xx.xx, xx.xx.xx.xx

nano /etc/hosts.deny

add this to the file and save:
sshd:ALL

This will stop ssh from answering to anyone except the IP's you list in hosts.allow

You can also change ssh's port it answers on, this file depends on the OS your running, ill assume centcrapOS
nano /etc/ssh/sshd_config

Change this:
# What ports, IPs and protocols we listen for
Port 22 to post 2200 or 2222 etc and save the faile.

/etc/rc.d/init.d/sshd restart ( You can restart it even when your logged in)