10.1. Remote Login and File Transfer¶
10.1.1. A Little History¶
The 1970’s
Before the days of Personal Computers (PCs), computers were giant room sized machines called Main Frames as we discussed in the `Introduction`_ to this course. These computers were large machines running in an isolated and temperature controlled place (usually called The Machine Room ) and users logged on to the Main Frame. These computers supported many users, logged in simultaneously.
The early 1980’s
Then came the PC. With a personal computer, the programs and files, etc. were all physically located in the box on the user’s desk, where everything is self contained. Typically, only one user would be logged on to the computer at any time.
The mid 1980’s
About the same time as the rise of the PC, the Internet (but not The Web which uses the Internet) became an important part of the worldwide cyberstructure allowing computers to transmit packets of data addressed to a unique computer somewhere else on the internet. Still The Web as you know it know did not exist. In the late 1980’s we had three basic forms of using the Internet:
- Telnet This is a way of logging into a remote computer, as if you were sitting at a terminal connected to that computer. Once logged in, you got a shell prompt on that machice, just like you get when you open an xterm on your computer. You could issue unix commands and move around the file tree from the prompt.
- FTP (File Transfer Protocol). This was a separate program from Telnet which allows users to transfer files to/from a remote computer.
In the days before the World Wide Web (WWW), this was the way we got things from the internet. There were archives of open files that one could access by using FTP protocol to login as “Anonymous” with password “your@email.address”. Once in, you could browse and download files—usually through a 9600 baud modem...
1990
Then came the World Wide Web, a system that linked files on different computers through ” Browsers that could interpret files written in “HyperText Markup Language” (HTML). These files could contain “Links” to files, or other HTML pages. This system was developed at the European Particle Accelerator, CERN. Recently the CERN Courier, the monthly news publication from CERN, published an article marking the 20th anniversary of the Web. It featured this document:
which is a copy of Tim Berners-Lee’s original memo to his boss outlining an idea for a system of servers, browsers, and interactions. Notice the handwritten note his boss wrote at the top: “Vague but exciting...”, scribbled by Berner-Lee’s boss.
The idea went on to transform the world and lead to The Information Age of global connectivity. The complete article for the CERN Courier is here if you are interested in reading it. While we can now cruise the Web and find a truly amazing amount of information in a variety of forms, there are still times when it is necessary to login to a remote machine and perform tasks from a non- graphical (no mouse or menus) prompt.
10.2. Get Connected with SSH¶
In the scientific world you usually remote login to other computers because
- the other computer has special resources that are not available on your computer. For example, the remote machine might be one of those supercomputers you looked at in Homework 1. It is super-fast and has a ridiculously large disk.
- your files are there. In this case you might use (S)FTP to get them
- Often certain computers at an institution or lab have certain software applications on them. One computer might have data visualization packages, another might have the statistical software. Another may be a computing cluster for running programs. Yet another could be the storage server connected to a large disk.
Thus we need to learn the basics of remote login.
10.2.1. SSH: Secure Shell¶
Telnet, which I mentioned above, is a program that you run at the command line of an xterm shell window. It opens a connection to a remote machine which allows access via telnet, then presents the user a login prompt (asks for your username and password) on the remote machine. If the user can provide the correct password, a shell prompt awaiting commands was returned.
The problem is... Telnet transactions are sent over the internet as plain text, i.e. as simple ASCII characters. These packets bounce from one router to the next to the next and so on, until they finally arrive at their destination. Anywhere along the way, an unscrupulous person could record packets traveling through nearby routers and read all the data. It is very simple to look for the characters ” Username: ” and ” Password: ” in the stream, then read the characters following this (up to the RETURN). The hacker would then have the person’s username and password.
Note
In about 2001 I discovered a breach of Pacific’s network by finding a hacker logged into one of the machines here. After killing his login and looking around a bit, I found that he had set up a program to log router traffic which left a file which contained the usernames and passwords of all the people on the south campus who accessed our school database via telnet on a router in the Psych building.
Because of this big flaw in Telnet, not many computers allow telnet access these days. The replacement for Telnet, is Secure Shell (SSH). Basically SSH is the same as Telnet–offering the user a shell prompt on the remote machine–however it encrypts all the data that is transmitted between the machines. Even if a hacker recorded the entire transmitted session, it would only look like giberish. The transmission can only be decoded by public key decryption, which is quite strong if the keys are sufficiently large.
10.2.2. Using SSH¶
First, we need a machine that we can ssh into.
I’ve made an account for us to use on physics-nix.stk.pacific.edu.
The username (for all of us) is phystu
The password for this account is given under the Resources course tool menu item on the Sakai PHYS 27 site (look for Resources on the left hand tool menu). Open the Resourses Folder and look for the document on the SSH Accounts. The reason for putting this here is that I don’t want to post the password to this account on the internet as a webpage. I only want those who have PHYS 27/193 access to Sakai to be able to view it.
Using what you learned above, ssh to physics-nix.stk.pacific.edu.
Remember that your USERNAME on this machine will be phystu (we will be sharing the account) and the domain name is given above (physics-nix.stk.pacific.edu).
Also,
Warning
You must be on campus to login to physics-nix. If you live off campus (outside the Pacific firewall), you will need to contact me about getting VPN (Virtual Private Network) access. Or you can just do the work of this chapter and the next by bringing your laptop on campus.
Exercise
Open a terminal shell, and type:
ssh phystu@physics-nix.stk.pacific.edu
Once you are logged into physics-nix.stk.pacific.edu you will get a prompt that looks like this:
phystu@physics-nix[~]>
I’ve preloaded the .bashrc file for you.
Do a directory listing
ls
to see what files are there (not much).
10.2.3. Emacs on a remote machine¶
Now that you have a shell prompt on the remote machine ( physics-nix), you can issue commands there.
For example, the which commmand tells you if a command or program is
available, and the location of the program file.
Type:
which gnuplot
to see the location of the gnuplot executable file on this machine.
Suppose we wanted to edit a file on physics-nix. You could type emacs, and in a little while an Emacs window will pop up on your screen. The reason it takes a little while is that the full X-Windowed data is being sent through the network. physics-nix must get information about your graphics capability, the location of other windows, the status of your mouse, etc., then send the information for an Emacs window though the internet to the machine you are currently working from (the laptop or desktop on which you are now working). AND–all this gets encrypted.
This is a fairly bulky process involving a lot of network traffic. Sometimes it’s necessary–say, if you are using gnuplot; you have to be able to look at the graph.
However, if you can save on bandwidth, your connection to the remote computer will be faster.
This is why it’s useful to be able to use emacs in “Text Mode” as we discussed when we learned in the Emacs: GUI v. Text.
You can start emacs in text mode on physics-nix by typing
emacs -nw
Remember this command? The -nw stands for “No Window”.
Now you can run programs and edit files on the remote machine. What more do you need!
You could be in a cyber cafe in Paris, ssh’ed into a computer here in the US, and–as long as you are familiar with moving around in a shell at the command prompt–have pretty much complete access and control of the computer here in the US.
In my research, I use supercomputers around the world, like Hopper shown here at NERSC
Once I am granted an account on one of these machines (usually by a grant writing procedure, where I get access for a certain amount of time), I can login via ssh from anywhere, just as you have done in this tutorial.
I then use emacs to create and modify files, such as simulation programs which I then run on that particular computer, making use of the fact that it is INCREDIBLY powerful and has a HUGE amount of disk space.
Once these programs have finished running, I can analyze the output on the supercomputer, or even better, reduce and transfer the data file to my local computer for analysis here (more on file transfer shortly).
Exercise
Before you log out from physics-nix, use emacs to create a file called “YOURNAME_nix.txt”. You will need this file below.
where YOURNAME is (duh), your name. In the file, put the text
This file was created by USERNAME on physics-nix.stk.pacific.edu.
Date: today's date
I'm feeling: --how are you feeling today?--
10.2.4. Getting Back Out¶
When you are finished with your work on the remote computer, you logout by typing:
logout
oddly enough.
Equivalently, you can type exit or even just: CTRL-d (this is my fave).
All of these do the same thing: exit your session on the remote machine and log you out.
Logout from physics-nix.stk.pacific.edu, if you haven’t already.
This should leave you at the prompt in your terminal shell on your laptop/computer.
The reason I set up your .bashrc file to display:
sci[~]>
in the propmt is so that you will know that you entering commands back on your local laptop. For the same reason, I put phystu@physics-nix[~]> in the .bashrc file for user phystu on physics-nix. It helps keep track of which machine you are logged into when typing commands at the prompt.
10.3. SFTP and SCP¶
Sometimes you don’t want a prompt on the remote machine to run commands,, but rather you need to get (or put) some files from (or to) the remote machine.
That’s when you call on sftp: Secure File Transfer Protocol. The syntax is very much like ssh.
10.3.1. SFTP¶
SFTP stands for Secure File Transfer Protocol.
At the prompt on your local machine (your laptop, say), type:
sftp phystu@physics-nix.stk.pacific.edu
In a short while, physics-nix.stk.pacific.edu will ask you for the password for the phystu account (this is the same one as you just used to ssh to physics-nix above. Finally, you will get a prompt that looks like this:
sftp>
You can find all the commands that sftp accepts by typing ? .
sftp> ?
Available commands:
cd path Change remote directory to 'path'
lcd path Change local directory to 'path'
chgrp grp path Change group of file 'path' to 'grp'
chmod mode path Change permissions of file 'path' to 'mode'
chown own path Change owner of file 'path' to 'own'
help Display this help text
get remote-path [local-path] Download file
lls [ls-options [path]] Display local directory listing
ln oldpath newpath Symlink remote file
lmkdir path Create local directory
lpwd Print local working directory
ls [path] Display remote directory listing
lumask umask Set local umask to 'umask'
mkdir path Create remote directory
put local-path [remote-path] Upload file
pwd Display remote working directory
exit Quit sftp
quit Quit sftp
rename oldpath newpath Rename remote file
rmdir path Remove remote directory
rm path Delete remote file
symlink oldpath newpath Symlink remote file
version Show SFTP version
!command Execute 'command' in local shell
! Escape to local shell
? Synonym for help
You can intuit most of these. The most commonly used commands are:
- cd path change directory on the remote machine
- lcd path change directory on the local machine
- get filename download a file from the remote to local machine
- put filename upload a file from the local to remote machine
- mget filen* download many files (* matches any characters )
- mput filen* upload many files (* matches any characters )
- quit or exit exit sftp
Exercise
Make sure you have opened an sftp session on physics-nix as described above.
Now, at the sftp> prompt, type
get USERNAME_nix.txt
You should see something like this:
sftp> get jhetrick_nix.txt
Fetching /home/jhetrick/jhetrick_nix.txt to /jhetrick.txt
/home/jhetrick//jhetrick_nix.txt 100% 444 0.4KB/s 00:00
Now quit the SFTP session, by typing.... quit.
Once you have your sci[~]> prompt back, indicating you are in the shell on your local computer,
do an ls to see that you indeed downloaded the file USERNAME.txt
from physics-nix to your laptop.
Is it there?
View the contents of the file (remember how?) and check that its contents say that it was created by you on physics-nix.
This is how you get file to and from remote computers.
To upload files, you do pretty much the same thing, except that you use the put command.
Exercise
On your local computer (i.e. your laptop), create a file called “USERNAME_local.txt”. Add the text “This file was created on my super-deluxe Dell-o-tron.” (or something similar that describes your laptop/computer) to the file, so that you know that this was the locally created file.
Upload this file to physics-nix.
I’ll be able to see this file on physics-nix so I’ll be able to tell if you have done it correctly.
This is part of Homework 10.
10.3.2. SCP¶
Another way to transfer files is to use Secure CP. This tool is a mashup of SFTP and the usual unix CP command.
Recall how we copy a file from one place to another, by using the cp command:
cp file1 file2
cp file1 dir/
cp file1 dir/newname
to:
- make a copy of file1 called file2, in the same directory
- make a copy of file1 in the directory dir/
- make a copy of file1 called newname in directory dir/
In general, the cp command takes the form:
cp source target
meaning that the first thing (the source) is copied to the second thing (the target).
SCP allows us to use essentially the same one-liner command, but include reference to files and directories on remote machines. The syntax goes like this:
sci[]> scp file1 janedoe@remote.mach.ine:dir1/dir2/file2
This command will
open a connection to the machine called remote.mach.ine and login as user janedoe and ask for her password on the remote machine.
This is what is meant by the text: janedoe@remote.mach.ine:
put a copy of file1 in the sub-subdirectory dir1/dir2 beneath janedoe’s HOME directory on the remote machine, called file2.
Note
if your username on the local machine AND the remote machine is the same, you don’t have
to include the username (janedoe, in this example). You could just give the remote machine
name. In that case, you would type scp file1 remote.mach.ine: to put file1 in your HOME
directory on remote.mach.ine ).
Exercise
To try this out for yourself, open a terminal shell on your laptop,
and make a test file (emacs, “blah blah”, save-> mynametest.txt).
Now scp this file to physics-nix. If the file you just made is in the current directory, you would do this:
scp mynametest.txt phystu@physics-nix.serv.pacific.edu:
Rememeber the password for phystu on physics-nix; you’ll be asked for it.
This should put a copy of the file mynametest.txt,
in the home directory of user: phystu, on the remote machine: physics-nix.serv.pacific.edu.
Now, logon to physics-nix.serv.pacific.edu using ssh:
ssh phystu@physics-nix.serv.pacific.edu
At the prompt on physics-nix, do a directory listing and verify that the file mynametest.txt is there.
Edit it with emacs (remotely, using text-mode emacs as discussed above), and add a sentence to the end:
“This file was successfully copied to physics-nix with scp“
Now, logout from physics-nix (type exit, or CTRL-d). You should have the sci[~]> prompt back in your
local terminal shell.
Delete the local mynametest.txt file (rm mynametest.txt).
We are going to grab back the copy we just put on physics-nix.
Do an ls to verify it is gone.
Now, get a copy from physics-nix:
scp phystu@physics-nix.serv.pacific.edu:mynametest.txt .
Notice this time the source is on the remote machine, and the target location/name is just . . Remember, ., a single period, means “here” in unix. It stands for “the current directory”.
This should prompt you for your password on the remote machine, then copy the requested file to the current directory on the local machine.
If you were successful, you should have a file on your local machine that has the added sentence: “This file was successfully copied to physics-nix with scp”. That’s how you know you have pulled the copy from physics-nix.serv.pacific.edu, after you put it there in the first place.
10.3.3. What’s the difference between sftp and scp?¶
Basically, sftp and scp do the same thing–move files from one computer to another. However, sftp is a bigger program, whereas scp is a oneliner that allows you to grab a single (or multiple) file(s), if you know where on the remote machine they are. You have to give scp “the full path/name” information to the file in order to find it.
With sftp, you get an sftp> prompt. SFTP really gives you a
“file transfer shell–an environment in which you have a prompt and
a number of commands that you can type, such as help, ls, cd
.., etc. This sftp shell allows you to move around within the
sftp environment (on the remote machine). So, if you can’t remember
where the file is located on the remote machine, sftp would be the
way to go. You can login to the remote machine, cd around, ls
to view the directory contents, and then grab the files you need with
the get command.
10.4. Homework¶
If you did the above exercises, you are ready for Homework 10, which is here: