Yes, if lynx (or another text based web browser) is installed on your server, you can download the archive directly to the server. Obviously keep it out of any area accessible to the web server.
And if you're so inclined, you can even run the upgrade script via the SSH command line, if your PHP is set to allow CLI.