The Unix Shell

At a high level, computers do four things:

• run programs;
• store data;
• communicate with each other; and
• interact with us.

They can do the last of these in many different ways, including direct brain-computer links and speech interfaces. Since these are still in their infancy, most of us use windows, icons, mice, and pointers. These technologies didn't become widespread until the 1980s, but their roots go back to Doug Engelbart's work in the 1960s, which you can see in what has been called "The Mother of All Demos".

Going back even further, the only way to interact with early computers was to rewire them. But in between, from the 1950s to the 1980s, most people used a technology based on the old-fashioned typewriter. That technology is still widely used today, and is the subject of this chapter.

When I say "typewriter", I actually mean a line printer connected to a keyboard (Figure 1). These devices only allowed input and output of the letters, numbers, and punctuation found on a standard keyboard, so programming languages and interfaces had to be designed around that constraint—though if you had enough time on your hands, you could draw simple pictures using just those characters (Figure 2).

                    ,-.             __
                  ,'   `---.___.---'  `.
                ,'   ,-                 `-._
              ,'    /                       \
           ,\/     /                        \\
       )`._)>)     |                         \\
       `>,'    _   \                  /       ||
         )      \   |   |            |        |\\
.   ,   /        \  |    `.          |        | ))
\`. \`-'          )-|      `.        |        /((
 \ `-`   .`     _/  \ _     )`-.___.--\      /  `'
  `._         ,'     `j`.__/           `.    \
    / ,    ,'         \   /`             \   /
    \__   /           _) (               _) (
      `--'           /____\             /____\

Today, this kind of interface is called a command-line user interface, or CLUI, to distinguish it from the graphical user interface, or GUI, that most of us now use. The heart of a CLUI is a read-evaluate-print loop, or REPL: when the user types a command, the computer reads it, executes it, and prints its output. (In the case of old teletype terminals, it literally printed the output onto paper, a line at a time.) The user then types another command, and so on until the user logs off.

This description makes it sound as though the user sends commands directly to the computer, and the computer sends output directly to the user. In fact, there is usually a program in between called a command shell (Figure 3). What the user types goes into the shell; it figures out what commands to run and orders the computer to execute them.

A shell is a program like any other. What's special about it is that its job is to run other programs, rather than to do calculations itself. The most popular Unix shell is Bash, the Bourne Again SHell (so-called because it's derived from a shell written by Stephen Bourne—this is what passes for wit among programmers). Bash is the default shell on most modern implementations of Unix, and in most packages that provide Unix-like tools for Windows, such as Cygwin.

Using Bash, or any other shell, sometimes feels more like programming that like using a mouse. Commands are terse (often only a couple of characters long), their names are frequently cryptic, and their output is lines of text rather than something visual like a graph. On the other hand, the shell allows us to combine existing tools in powerful ways with only a few keystrokes, and to set up pipelines to handle large volumes of data automatically. In addition, the command line is often the easiest way to interact with remote machines. As clusters and cloud computing become more popular for scientific data crunching, being able to drive them is becoming a necessary skill.

The part of the operating system responsible for managing files and directories is called the file system. It organizes our data into files, which hold information, and directories (also called "folders"), which hold files or other directories.

Several commands are frequently used to create, inspect, rename, and delete files and directories. To start exploring them, let's log in to the computer by typing our user ID and password. Most systems will print stars to obscure the password, or nothing at all, in case some evildoer is shoulder surfing behind us.

login: vlad
password: ********
$

Once we have logged in we'll see a prompt, which is how the shell tells us that it's waiting for input. This is usually just a dollar sign, but which may show extra information such as our user ID or the current time. Type the command whoami, then press the Enter key (sometimes marked Return) to send the command to the shell. The command's output is the ID of the current user, i.e., it shows us who the shell thinks we are:

$ whoami
vlad
$

More specifically, when we type whoami the shell:

1. finds a program called whoami,
2. runs it,
3. waits for it to display its output, and
4. displays a new prompt to tell us that it's ready for more commands.

Getting Around

Next, let's find out where we are by running a command called pwd (which stands for "print working directory"). At any moment, our current working directory is our current default directory, i.e., the directory that the computer assumes we want to run commands in unless we explicitly specify something else. Here, the computer's response is /users/vlad, which is Vlad's home directory:

$ pwd
/users/vlad
$

To understand what a "home directory" is, let's have a look at how the file system as a whole is organized (Figure 4). At the top is the root directory that holds everything else the computer is storing. We refer to it using a slash character / on its own; this is the leading slash in /users/vlad.

Inside that directory (or underneath it, if you're drawing a tree) are several other directories: bin (which is where some built-in programs are stored), data (for miscellaneous data files), users (where users' personal directories are located), tmp (for temporary files that don't need to be stored long-term), and so on. We know that our current working directory /users/vlad is stored inside /users because /users is the first part of its name. Similarly, we know that /users is stored inside the root directory / because its name begins with /.

Underneath /users, we find one directory for each user with an account on this machine (Figure 5). The Mummy's files are stored in /users/imhotep, Wolfman's in /users/larry, and ours in /users/vlad, which is why vlad is the last part of the directory's name. Notice, by the way, that there are two meanings for the / character. When it appears at the front of a file or directory name, it refers to the root directory. When it appears inside a name, it's just a separator.

Let's see what's in Vlad's home directory by running ls, which stands for "listing":

$ ls
bin          data      mail       music
notes.txt    papers    pizza.cfg  solar
solar.pdf    swc
$

ls prints the names of the files and directories in the current directory in alphabetical order, arranged neatly into columns. To make its output more comprehensible, we can give it the flag -F by typing ls -F. This tells ls to add a trailing / to the names of directories:

$ ls -F
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/
$

As you can see, /users/vlad contains seven sub-directories (Figure 6). The names that don't have trailing slashes—notes.txt, pizza.cfg, and solar.pdf—are plain old files.

Now let's take a look at what's in Vlad's data directory by running the command ls -F data. The second parameter—the one without a leading dash—tells ls that we want a listing of something other than our current working directory:

$ ls -F data
amino_acids.txt   elements/     morse.txt
pdb/              planets.txt   sunspot.txt
$

The output shows us that there are four text files and two sub-sub-directories. Organizing things hierarchically in this way is a good way to keep track of our work: it's possible to put hundreds of files in our home directory, just as it's possible to pile hundreds of printed papers on our desk, but in the long run it's a self-defeating strategy.

Notice, by the way that we spelled the directory name data. It doesn't have a trailing slash: that's added to directory names by ls when we use the -F flag to help us tell things apart. And it doesn't begin with a slash because it's a relative path, i.e., it tells ls how to find something from where we are, rather than from the root of the file system (Figure 7).

If we run ls -F /data (with a leading slash) we get a different answer, because /data is an absolute path:

$ ls -F /data
access.log    backup/    hardware.cfg
network.cfg
$

The leading / tells the computer to follow the path from the root of the filesystem, so it always refers to exactly one directory, no matter where we are when we run the command.

What if we want to change our current working directory? Before we do this, pwd shows us that we're in /users/vlad, and ls without any parameters shows us that directory's contents:

$ pwd
/users/vlad
$ ls
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/
$

We can use cd followed by a directory name to change our working directory. cd stands for "change directory", which is a bit misleading: the command doesn't change the directory, it changes the shell's idea of what directory we are in.

$ cd data
$

cd doesn't print anything, but if we run pwd after it, we can see that we are now in /users/vlad/data. If we run ls without parameters now, it lists the contents of /users/vlad/data, because that's where we now are:

$ pwd
/users/vlad/data
$ ls
amino_acids.txt   elements/     morse.txt
pdb/              planets.txt   sunspot.txt
$

OK, we can go down the directory tree: how do we go up? We could use an absolute path:

$ cd /users/vlad
$

but it's almost always simpler to use cd .. to go up one level:

$ pwd
/users/vlad/data
$ cd ..

.. is a special directory name meaning "the directory containing this one", or, more succinctly, the parent of the current directory.

Sure enough, if we run pwd after running cd .., we're back in /users/vlad:

$ pwd
/users/vlad
$

The special directory .. doesn't usually show up when we run ls. If we want to display it, we can give ls the -a flag:

$ ls -F -a
./           ../       bin/       data/
mail/        music/    notes.txt  papers/
pizza.cfg    solar/    solar.pdf  swc/

-a stands for "show all"; it forces ls to show us file and directory names that begin with ., such as .. (which, if we're in /users/vlad, refers to the /users directory). As you can see, it also displays another special directory that's just called ., which means "the current working directory". It may seem redundant to have a name for it, but we'll see some uses for it soon.

Everything we have seen so far works on Unix and its descendents, such as Linux and Mac OS X. Things are a bit different on Windows. A typical directory path on a Windows 7 machine might be C:\Users\vlad. The first part, C:, is a drive letter that identifies which disk we're talking about. This notation dates back to the days of floppy drives; today, different "drives" are usually different filesystems on the network.

Instead of a forward slash, Windows uses a backslash to separate the names in a path. This causes headaches because Unix uses backslash for input of special characters. For example, if we want to put a space in a filename on Unix, we would write the filename as my\ results.txt. Please don't ever do this, though: if you put spaces, question marks, and other special characters in filenames on Unix, you can confuse the shell for reasons that we'll see shortly.

Finally, Windows filenames and directory names are case insensitive: upper and lower case letters mean the same thing. This means that the path name C:\Users\vlad could be spelled c:\users\VLAD, C:\Users\Vlad, and so on. Some people argue that this is more natural: after all, "VLAD" in all upper case and "Vlad" spelled normally refer to the same person. However, it causes headaches for programmers, and can be difficult for people to understand if their first language doesn't use a cased alphabet.

$ ls north-pacific-gyre/2012-07-03/

$ ls no

$ ls north-pacific-gyre/

We now know how to explore files and directories, but how do we create them in the first place? Let's go back to Vlad's home directory, /users/vlad, and use ls -F to see what it contains:

$ pwd
/users/vlad
$ ls -F
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/
$

Let's create a new directory called thesis using the command mkdir thesis (which has no output):

$ mkdir thesis
$

As you might (or might not) guess from its name, mkdir means "make directory". Since thesis is a relative path (i.e., doesn't have a leading slash), the new directory is made below the current one:

$ ls -F
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/      thesis/
$

However, there's nothing in it yet:

$ ls -F thesis
$

Let's change our working directory to thesis using cd, then run an editor called Nano to create a file called nano draft.txt:

$ cd thesis
$ nano draft.txt

Nano is a very simple text editor that only a programmer could really love. Figure 9 shows what it looks like when it runs: the cursor is the blinking square in the upper left, and the two lines across the bottom show us the available commands. (By convention, Unix documentation uses the caret ^ followed by a letter to mean "control plus that letter", so ^O means Control+O.)

Let's type in a short quotation (Figure 10) then use Control-O to write our data to disk. Once our quotation is saved, we can use Control-X to quit the editor and return to the shell.

nano doesn't leave any output on the screen after it exits. But ls now shows that we have created a file called draft.txt:

$ ls
draft.txt
$

If we just want to quickly create a file with no content, we can use the command touch.

$ touch draft_empty.txt
$ ls
draft.txt   draft_empty.txt
$

The touch command has allowed us to create a file without leaving the command line. As with text editors like nano, touch will create the file in your current working directory.

If we touch a file that already exists, the time of last modification for the file is updated to the present time, but the file is otherwise unaltered. We can demonstrate this by running ls with the -l flag to give a long listing of files in the current directory which includes information about the date and time of last modification.

$ ls -l
-rw-r--r-- 1 username username 512 Oct  9 15:46 draft.txt
-rw-r--r-- 1 username username   0 Oct  9 15:47 draft_empty.txt
$ touch draft.txt
$ ls -l
-rw-r--r-- 1 username username 512 Oct  9 15:48 draft.txt
-rw-r--r-- 1 username username   0 Oct  9 15:47 draft_empty.txt

Note how the modification time of draft.txt changes after we touch it.

We can run ls with the -s flag (for "size") to show us how large draft.txt is:

$ ls -s
   1  draft.txt        0  draft_empty.txt
$

Unfortunately, Unix reports sizes in disk blocks by default, which might be the least helpful default imaginable. If we add the -h flag, ls switches to more human-friendly units:

$ ls -s -h
 512  draft.txt        0  draft_empty.txt
$

Here, 512 is the number of bytes in the draft.txt file. This is more than we actually typed in because the smallest unit of storage on the disk is typically a block of 512 bytes.

Let's start tidying up by running rm draft.txt:

$ rm draft.txt
$

This command removes files ("rm" is short for "remove"). If we run ls, only the draft_empty.txt file remains:

$ ls
draft_empty.txt
$

Now we'll finish cleaning up by running rm draft_empty.txt:

$ rm draft_empty.txt
$

If we run ls again, its output is empty, which tells us that both of our files are gone:

$ ls
$

Let's re-create draft.txt as an empty file, and then move up one directory to /users/vlad using cd ..:

$ pwd
/users/vlad/thesis
$ touch draft.txt
$ ls
draft.txt
$ cd ..
$

If we try to remove the entire thesis directory using rm thesis, we get an error message:

$ rm thesis
rm: cannot remove `thesis': Is a directory
$

This happens because rm only works on files, not directories. The right command is rmdir, which is short for "remove directory": It doesn't work yet either, though, because the directory we're trying to remove isn't empty:

$ rmdir thesis
rmdir: failed to remove `thesis': Directory not empty
$

This little safety feature can save you a lot of grief, particularly if you are a bad typist. If we really want to get rid of thesis we should first delete the file draft.txt:

$ rm thesis/draft.txt
$

The directory is now empty, so rmdir can delete it:

$ rmdir thesis
$

$ rm -r thesis
$

Let's create that directory and file one more time. (Note that this time we're running touch with the path thesis/draft.txt, rather than going into the thesis directory and running touch on draft.txt there.)

$ pwd
/users/vlad/thesis
$ mkdir thesis
$ touch thesis/draft.txt
$ ls thesis
draft.txt
$

draft.txt isn't a particularly informative name, so let's change the file's name using mv, which is short for "move":

$ mv thesis/draft.txt thesis/quotes.txt
$

The first parameter tells mv what we're "moving", while the second is where it's to go. In this case, we're moving thesis/draft.txt to thesis/quotes.txt, which has the same effect as renaming the file. Sure enough, ls shows us that thesis now contains one file called quotes.txt:

$ ls thesis
quotes.txt
$

Just for the sake of inconsistency, mv also works on directories—there is no separate mvdir command.

Let's move quotes.txt into the current working directory. We use mv once again, but this time we'll just use the name of a directory as the second parameter to tell mv that we want to keep the filename, but put the file somewhere new. (This is why the command is called "move".) In this case, the directory name we use is the special directory name . that we mentioned earlier):

$ mv thesis/quotes.txt .
$

The effect is to move the file from the directory it was in to the current directory. ls now shows us that thesis is empty, and that quotes.txt is in our current directory:

$ ls thesis
$ ls quotes.txt
quotes.txt
$

(Notice that ls with a filename or directory name as an parameter only lists that file or directory.)

The cp command works very much like mv, except it copies a file instead of moving it. We can check that it did the right thing using ls with two paths as parameters—like most Unix commands, ls can be given thousands of paths at once:

$ cp quotes.txt thesis/quotations.txt
$ ls quotes.txt thesis/quotations.txt
quotes.txt   thesis/quotations.txt
$

To prove that we made a copy, let's delete the quotes.txt file in the current directory, and then run that same ls again. This time, it tells us that it can't find quotes.txt in the current directory, but it does find the copy in thesis that we didn't delete:

$ ls quotes.txt thesis/quotations.txt
ls: cannot access quotes.txt: No such file or directory
thesis/quotations.txt
$

At this point we have a basic toolkit which allows us to do things in the shell that you can normally do in Windows, Mac OS, or GNU/Linux with your mouse. For example, we can create, delete and move individual files. We will now learn about wildcards, which will allow us to expand our toolkit so that we can perform operations on multiple files with a single command. We'll start in an empty directory and create four files:

$ touch file1.txt file2.txt otherfile1.txt otherfile2.txt
$ ls
file1.txt   file2.txt   otherfile1.txt   otherfile2.txt
$

If we want to see all filenames that start with "file", we can run the command ls file*:

$ ls file*
file1.txt   file2.txt
$

There are four other basic wildcard characters we will explore:

• The question mark, ?, matches any single character, so text?.txt would match text1.txt or textz.txt, but not text_1.txt.
• The square brackets, [ and ], which can be used to specify a set of possible characters. For example, text[135].txt would match text1.txt, text3.txt, and text5.txt. You can also specify a natural sequence of characters using a dash. For example, text[1-6].txt would match text1.txt, text2.txt, ..., text6.txt.
• The exclamation mark, !, which can be used with square brackets to negate characters. For example, text[!135].txt would match anything of the form text?.txt except for text1.txt, text3.txt, and text5.txt. (Note that the exclamation mark has to be used right after the first square bracket as in the example. An exclamation mark outside of square brackets is not a wildcard, but serves another function of accessing previous commands from your history, as we will later see.)
• The curly braces, {}, are similar to the square brackets, except that they take a list of items separated by commas, and that they always expand all elements of the list, whether or not the corresponding filenames exist. For example, text{1,5}.txt would expand to text1.txt text5.txt whether or not text1.txt and text5.txt actually exist. The elements of the list need not be single characters, so text{12,5}.txt expands to text12.txt text5.txt. Natural sequences can be specified using .., i.e. text{1..5}.txt would expand to text1.txt text2.txt text3.txt text4.txt text5.txt.

We can use any number of wildcards at a time: for example, p*.p?* matches anything that starts with a 'p' and ends with '.', 'p', and at least one more character (since the '?' has to match one character, and the final '*' can match any number of characters). Thus, p*.p?* would match preferred.practice, and even p.pi (since the first '*' can match no characters at all), but not quality.practice (doesn't start with 'p') or preferred.p (there isn't at least one character after the '.p').

Let's see some more examples of wildcards in action. First we will remove everything in the current directory:

$ rm *
$ ls
$

Now that our working directory is empty, let's create some empty files using the touch command and the curly braces:

$ touch file{4,6,7}.txt file_{a..e}.txt
$ 

The curly braces expand the second token file{4,6,7}.txt into three separate file names, and the third token file_{a..e}.txt expands into five separate file names. So, now if we list files using the asterisk wildcard, we get:

$ ls file*
file4.txt   file7.txt    file_b.txt   file_d.txt
file6.txt   file_a.txt   file_c.txt   file_e.txt
$

We have matched all files in our current directory that begin with file. Now, let us list a subset of those files using the question mark:

$ ls file?.txt
file4.txt   file6.txt    file7.txt
$

Note how this glob pattern matched only the numbered files. That's because only the numbered files have a single character between file and .txt. The lettered files have two characters between file and .txt:

$ ls file??.txt
file_a.txt   file_b.txt    file_c.txt   file_d.txt   file_e.txt
$

Let's try out using the ! wildcard with square brackets. The pattern file[!46]* would match everything beginning with "file", unless the next character is a 4 or a 6. Thus, continuing with our example, if we do rm file[!46]*, we will delete everything except file4.txt and file6.txt:

$ ls
file4.txt   file7.txt    file_b.txt   file_d.txt
file6.txt   file_a.txt   file_c.txt   file_e.txt
$ rm file[!46]*
$ ls
file4.txt   file6.txt
$

It's also possible to combine and nest wildcards. For example, file{4,_?}.txt would match file4.txt and file_?.txt, where ? matches any single character. To demonstrate this, let's re-create all of our files:

$ touch file{4,6,7}.txt file_{a..e}.txt
$ ls
file4.txt   file7.txt    file_b.txt   file_d.txt
file6.txt   file_a.txt   file_c.txt   file_e.txt
$ 

Now let's run ls file{4,_?}.txt:

$ ls file{4,_?}.txt
file4.txt   file_b.txt   file_d.txt
file_a.txt   file_c.txt   file_e.txt
$

Finally, it is important to understand what happens if a wildcard expression doesn't match any files. In this case, the wildcard characters are interpreted literally. For example, if we were to enter ls file[AB].txt, the system will first try to find fileA.txt or fileB.txt. If it found either of those, then the wildcard expression would be expanded appropriately, but when it fails to find either of them, the expression remains unevaluated as file[AB].txt, and is then passed to ls. At this point, ls will try to find a single file that is actually named file[AB].txt. We can see that this is the case if we try ls file[AB].txt:

$ ls file[AB].txt
ls: file[AB].txt: No such file or directory
$

The system tells us that file[AB].txt doesn't exist since this was the file it was looking for after the wildcard expression found no matches. Still not convinced? Let's actually create a file named file[AB].txt. To do this, we need some way of telling the computer that we actually want the [ and ] characters to be in the filename, and that we don't mean them as wildcards.

Now we can create the file file[AB].txt:

$ touch "file[AB].txt"
$ ls
file4.txt   file7.txt      file_a.txt   file_c.txt   file_e.txt
file6.txt   file[AB].txt   file_b.txt   file_d.txt
$

If we do ls file[AB].txt, it will find the file we've just created:

$ ls file[AB].txt
file[AB].txt
$

But if we create a file named fileA.txt, this will be found first by the wildcard expansion, since it first looks for fileA.txt and fileB.txt:

$ touch fileA.txt
$ ls file[AB].txt
fileA.txt
$

If we really do want to look for something called file[AB].txt, and not fileA.txt or fileB.txt, then we have to escape the special characters "[" and "]". We can do this:

$ ls file\[AB\].txt
file[AB].txt
$

or this:

$ ls "file[AB].txt"
file[AB].txt
$

Single quotes would work as well in this case.

Now that we know a few basic commands and have learned how to use wildcards, we can finally look at the shell's most powerful feature: the ease with which it lets you combine existing programs in new ways. We'll start with a directory called molecules that contains six files describing some simple organic molecules. The .pdb extension indicates that these files are in Protein Data Bank format, a simple text format that specifies the type and position of each atom in the molecule.

$ ls molecules
cubane.pdb    ethane.pdb    methane.pdb
octane.pdb    pentane.pdb   propane.pdb
$

Let's go into that directory with cd and run the command wc *.pdb. wc is the "word count" command; it counts the number of lines, words, and characters in files:

$ cd molecules
$ wc *.pdb
  20  156 1158 cubane.pdb
  12   84  622 ethane.pdb
   9   57  422 methane.pdb
  30  246 1828 octane.pdb
  21  165 1226 pentane.pdb
  15  111  825 propane.pdb
 107  819 6081 total
$

If we run wc -l instead of just wc, the output shows only the number of lines per file:

$ wc -l *.pdb
  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total
$

We can also use -w to get only the number of words, or -c to get only the number of characters.

Now, which of these files is shortest? It's an easy question to answer when there are only six files, but what if there were 6000? That's the kind of job we want a computer to do.

Our first step toward a solution is to run the command:

$ wc -l *.pdb > lengths

The > tells the shell to redirect the command's output to a file instead of printing it to the screen. The shell will create the file if it doesn't exist, or overwrite the contents of that file if it does. (This is why there is no screen output: everything that wc would have printed has gone into the file lengths instead.)

ls lengths confirms that the file exists:

$ ls lengths
lengths
$

We can now send the content of lengths to the screen using cat lengths. cat stands for "concatenate": it prints the contents of files one after another. In this case, there's only one file, so cat just shows us what it contains:

$ cat lengths
  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total
$

Now let's use the sort command to sort its contents. This does not change the file. Instead, it sends the sorted result to the screen:

$ sort lengths
  9  methane.pdb
 12  ethane.pdb
 15  propane.pdb
 20  cubane.pdb
 21  pentane.pdb
 30  octane.pdb
107  total
$

We can put the sorted list of lines in another temporary file called sorted-lengths by putting > sorted-lengths after the command, just as we used > lengths to put the output of wc into lengths. Once we've done that, we can run another command called head to get the first few lines in sorted-lengths:

$ sort lengths > sorted-lengths
$ head -1 sorted-lengths
  9  methane.pdb
$

Giving head the parameter -1 tells us we only want the first line of the file; -20 would get the first 20, and so on. Since sorted-lengths the lengths of our files ordered from least to greatest, the output of head must be the file with the fewest lines.

If you think this is confusing, you're in good company: even once you understand what wc, sort, and head do, all those intermediate files make it hard to follow what's going on. How can we make it easier to understand?

Let's start by getting rid of the sorted-lengths file by running sort and head together:

$ sort lengths | head -1
  9  methane.pdb
$

The vertical bar between the two commands is called a pipe. It tells the shell that we want to use the output of the command on the left as the input to the command on the right. The computer might create a temporary file if it needs to, or copy data from one program to the other in memory, or something else entirely: we don't have to know or care.

Well, if we don't need to create the temporary file sorted-lengths, can we get rid of the lengths file too? The answer is "yes": we can use another pipe to send the output of wc directly to sort, which then sends its output to head:

$ wc -l *.pdb | sort | head -1
  9  methane.pdb
$

This is exactly like a mathematician nesting functions like sin(πx)² and saying "the square of the sine of x times π": in our case, the calculation is "head of sort of word count of *.pdb".

Inside Pipes

Here's what actually happens behind the scenes when we create a pipe. When a computer runs a program—any program—it creates a process in memory to hold the program's software and its current state. Every process has an input channel called standard input. (By this point, you may be surprised that the name is so memorable, but don't worry: most Unix programmers call it stdin.) Every process also has a default output channel called standard output, or stdout (Figure 11).

The shell is just another program, and runs in a process like any other. Under normal circumstances, whatever we type on the keyboard is sent to the shell on its standard input, and whatever it produces on standard output is displayed on our screen (Figure 12):

When we run a program, the shell creates a new process. It then temporarily sends whatever we type on our keyboard to that process's standard input, and whatever the process sends to standard output to the screen (Figure 13):

Here's what happens when we run wc -l *.pdb > lengths. The shell starts by telling the computer to create a new process to run the wc program. Since we've provided some filenames as parameters, wc reads from them instead of from standard input. And since we've used > to redirect output to a file, the shell connects the process's standard output to that file (Figure 14).

If we run wc -l *.pdb | sort instead, the shell creates two processes, one for each process in the pipe, so that wc and sort run simultaneously. The standard output of wc is fed directly to the standard input of sort; since there's no redirection with >, sort's output goes to the screen (Figure 15):

And if we run wc -l *.pdb | sort | head -1, we get the three processes shown in Figure 16 with data flowing from the files, through wc to sort, and from sort through head to the screen.

This simple idea is why Unix has been so successful. Instead of creating enormous programs that try to do many different things, Unix programmers focus on creating lots of simple tools that each do one job well, and work well with each other. Ten such tools can be combined in 100 ways, and that's only looking at pairings: when we start to look at pipes with multiple stages, the possibilities are almost uncountable.

This programming model is called pipes and filters. We've already seen pipes; a filter is a program that transforms a stream of input into a stream of output. Almost all of the standard Unix tools can work this way: unless told to do otherwise, they read from standard input, do something with what they've read, and write to standard output.

The key is that any program that reads lines of text from standard input, and writes lines of text to standard output, can be combined with every other program that behaves this way as well. You can and should write your programs this way, so that you and other people can put those programs into pipes to multiply their power.

$ cd north-pacific-gyre/2012-07-03
$ wc -l *.txt

 300 NENE01729A.txt
 300 NENE01729B.txt
 300 NENE01736A.txt
 300 NENE01751A.txt
 300 NENE01751B.txt
 300 NENE01812A.txt
 ... ...

$ wc -l *.txt | sort | head -5
 240 NENE02018B.txt
 300 NENE01729A.txt
 300 NENE01729B.txt
 300 NENE01736A.txt
 300 NENE01751A.txt

$ wc -l *.txt | sort | tail -5
 300 NENE02040A.txt
 300 NENE02040B.txt
 300 NENE02040Z.txt
 300 NENE02043A.txt
 300 NENE02043B.txt

$ ls *Z.txt
NENE01971Z.txt    NENE02040Z.txt

Wildcards and tabs are one way to save on typing. Another is to tell the shell to do something over and over again. Suppose we have several hundred genome data files in a directory with names like basilisk.dat, unicorn.dat, and so on. When new files arrive, we'd like to rename the existing ones to original-basilisk.dat, original-unicorn.dat, etc. We can't use:

mv *.dat original-*.dat

because that would expand (in the two-file case) to:

mv basilisk.dat unicorn.dat

This wouldn't back up our files: it would replace the content of unicorn.dat with whatever's in basilisk.dat.

Instead, we can use a loop to do some operation once for each thing in a list. Here's a simple example that displays the first three lines of each file in turn:

$ for filename in basilisk.dat unicorn.dat
> do
>    head -3 $filename
> done
COMMON NAME: basilisk
CLASSIFICATION: basiliscus vulgaris
UPDATED: 1745-05-02
COMMON NAME: unicorn
CLASSIFICATION: equus monoceros
UPDATED: 1738-11-24

When the shell sees the keyword for, it knows it is supposed to repeat a command (or group of commands) once for each thing in a list. In this case, the list is the two filenames. Each time through the loop, the name of the thing currently being operated on is assigned to the variable filename. Inside the loop, we get the variable's value by putting $ in front of it (this is known as shell parameter expansion), so the first time through the loop, $filename is basilisk.dat, and the second time, unicorn.dat. Finally, the command that's actually being run is our old friend head, so this loop prints out the first three lines of each data file in turn.

for x in basilisk.dat unicorn.dat
do
    head -3 $x
done

for temperature in basilisk.dat unicorn.dat
do
    head -3 $temperature
done

Here's a slightly more complicated loop:

for filename in *.dat
do
    echo $filename
    head -100 $filename | tail -20
done

The shell starts by expanding *.dat to create the list of files it will process. The loop body then executes two commands for each of those files. The first, echo, just prints its command-line parameters to standard output. For example:

echo hello there

prints:

hello there

In this case, since the shell expands $filename to be the name of a file, echo $filename just prints the name of the file. Note that we can't write this as:

for filename in *.dat
do
    $filename
    head -100 $filename | tail -20
done

because then the first time through the loop, when $filename expanded to basilisk.dat, the shell would try to run basilisk.dat as a program. Finally, the head and tail combination selects lines 80-100 from whatever file is being processed.

basilisk.dat
red dragon.dat
unicorn.dat

for filename in *.dat
do
    echo $filename
    head -100 $filename | tail -20
done

basilisk.dat red dragon.dat unicorn.dat

basilisk.dat
red
dragon.dat
unicorn.dat

Going back to our original file renaming problem, we can solve it using this loop:

for filename in *.dat
do
    mv $filename original-$filename
done

This loop runs the mv command once for each filename. The first time, when $filename expands to basilisk.dat, the shell executes:

mv basilisk.dat original-basilisk.dat

The second time, the command is:

mv unicorn.dat original-unicorn.dat

for filename in *.dat
do
    echo mv $filename original-$filename
done

mv basilisk.dat original-basilisk.dat
mv unicorn.dat original-unicorn.dat

So far we have been renaming files of the form someanimal.dat to original-someanimal.dat. But, what if instead of original-someanimal.dat we want someanimal.dat.original or someanimal-original.dat? For someanimal.dat.original we can do

for filename in *.dat
do
    mv $filename $filename.original
done

But for someanimal-original.dat we will need to go a little deeper in shell parameter expansion and use its find and replace feature. That syntax is ${variable/find/substitute/} where find is the longest string in variable value that we want to be replaced by substitute. We can achieve this task using

for filename in *.dat
do
    mv $filename ${filename/.dat/-original.dat/}
done

$ cd north-pacific-gyre/2012-07-03
$ for datafile in *[AB].txt
do
    echo $datafile
done
NENE01729A.txt
NENE01729B.txt
NENE01736A.txt
...
NENE02043A.txt
NENE02043B.txt
$

$ for datafile in *[AB].txt
do
    echo $datafile stats-$datafile
done
NENE01729A.txt stats-NENE01729A.txt
NENE01729B.txt stats-NENE01729B.txt
NENE01736A.txt stats-NENE01736A.txt
...
NENE02043A.txt stats-NENE02043A.txt
NENE02043B.txt stats-NENE02043B.txt
$

$ for datafile in *[AB].txt; do echo $datafile stats-$datafile; done

$ for datafile in *[AB].txt; do goostats $datafile stats-$datafile; done

$ for datafile in *[AB].txt; do echo $datafile; goostats $datafile stats-$datafile; done

NENE01729A.txt
NENE01729B.txt
NENE01736A.txt
...
$

$ $ history | tail -5
  456  ls -l NENE0*.txt
  457  rm stats-NENE01729B.txt.txt
  458  goostats NENE01729B.txt stats-NENE01729B.txt
  459  ls -l NENE0*.txt
  460  history

We are finally ready to see what makes the shell such a powerful programming environment. We are going to take the commands we repeat frequently and save them in files so that we can re-run all those operations again later by typing a single command. For historical reasons, a bunch of commands saved in a file is usually called a shell script, but make no mistake: these are actually small programs.

Let's start by putting the following line in the file middle.sh:

head -20 cholesterol.pdb | tail -5

This is a variation on the pipe we constructed earlier: it selects lines 15-20 of the file cholesterol.pdb. Remember, we are not running it as a command just yet: we are putting the commands in a file.

Once we have saved the file, we can ask the shell to execute the commands it contains. Our shell is called bash, so we run the following command:

$ bash middle.sh
ATOM     14  C           1      -1.463  -0.666   1.001  1.00  0.00
ATOM     15  C           1       0.762  -0.929   0.295  1.00  0.00
ATOM     16  C           1       0.771  -0.937   1.840  1.00  0.00
ATOM     17  C           1      -0.664  -0.610   2.293  1.00  0.00
ATOM     18  C           1      -4.705   2.108  -0.396  1.00  0.00

Sure enough, our script's output is exactly what we would get if we ran that pipeline directly.

What if we want to select lines from an arbitrary file? We could edit middle.sh each time to change the filename, but that would probably take longer than just retyping the command. Instead, let's edit middle.sh and replace cholesterol.pdb with a special variable called $1:

$ cat middle.sh
head -20 $1 | tail -5

Inside a shell script, $1 means "the first filename (or other parameter) on the command line". We can now run our script like this:

$ bash middle.sh cholesterol.pdb
ATOM     14  C           1      -1.463  -0.666   1.001  1.00  0.00
ATOM     15  C           1       0.762  -0.929   0.295  1.00  0.00
ATOM     16  C           1       0.771  -0.937   1.840  1.00  0.00
ATOM     17  C           1      -0.664  -0.610   2.293  1.00  0.00
ATOM     18  C           1      -4.705   2.108  -0.396  1.00  0.00

or on a different file like this:

$ bash middle.sh vitamin_a.pdb
ATOM     14  C           1       1.788  -0.987  -0.861
ATOM     15  C           1       2.994  -0.265  -0.829
ATOM     16  C           1       4.237  -0.901  -1.024
ATOM     17  C           1       5.406  -0.117  -1.087
ATOM     18  C           1      -0.696  -2.628  -0.641

We still need to edit middle.sh each time we want to adjust the range of lines, though. Let's fix that by using the special variables $2 and $3:

$ cat middle.sh
head $2 $1 | tail $3
$ bash middle.sh vitamin_a.pdb -20 -5
ATOM     14  C           1       1.788  -0.987  -0.861
ATOM     15  C           1       2.994  -0.265  -0.829
ATOM     16  C           1       4.237  -0.901  -1.024
ATOM     17  C           1       5.406  -0.117  -1.087
ATOM     18  C           1      -0.696  -2.628  -0.641

What if we want to process many files in a single pipeline? For example, if we want to sort our PDB files by length, we would type:

$ wc -l *.pdb | sort -n

because wc -l lists the number of lines in the files, and sort -n sorts things numerically. We could put this in a file, but then it would only ever sort a list of PDB files in the current directory. If we want to be able to get a sorted list of other kinds of files, possibly from many sub-directories, we need a way to get all those names into the script. We can't use $1, $2, and so on because we don't know how many files there are. Instead, we use the special variable $*, which means, "All of the command-line parameters to the shell script." Here's an example:

$ cat sorted.sh
wc -l $* | sort -n
$ bash sorted.sh *.dat backup/*.dat
      29 chloratin.dat
      89 backup/chloratin.dat
      91 sphagnoi.dat
     156 sphag2.dat
     172 backup/sphag-merged.dat
     182 girmanis.dat

$ bash sorted.sh

wc -l | sort -n

sort $* | uniq

$ head -5 *.dat | bash distinct.sh

$ head -5 *.dat | sort | uniq

We have two more things to do before we're finished with our simple shell scripts. If you look at a script like:

wc -l $* | sort -n

you can probably puzzle out what it does. On the other hand, if you look at this script:

# List files sorted by number of lines.
wc -l $* | sort -n

you don't have to puzzle it out—the comment at the top of the script tells you what it does. A line or two of documentation like this make it much easier for other people (including your future self) to re-use your work. The only caveat is that each time you modify the script, you should check that the comment is still accurate: an explanation that sends the reader in the wrong direction is worse than none at all.

Second, suppose we have just run a series of commands that did something useful—for example, that created a graph we'd like to use in a paper. We'd like to be able to re-create the graph later if we need to, so we want to save the commands in a file. Instead of typing them in again (and potentially getting them wrong), we can do this:

$ history | tail -4 > redo-figure-3.sh

The file redo-figure-3.sh now contains:

 297 goostats -r NENE01729B.txt stats-NENE01729B.txt
 298 goodiff stats-NENE01729B.txt /data/validated/01729.txt > 01729-differences.txt
 299 cut -d ',' -f 2-3 01729-differences.txt > 01729-time-series.txt
 300 ygraph --format scatter --color bw --borders none 01729-time-series.txt figure-3.png

After a moment's work in an editor to remove the serial numbers on the command, we have a completely accurate record of how we created that figure.

In practice, most people develop shell scripts by running commands at the shell prompt a few times to make sure they're doing something useful, and doing it correctly, then saving them in a file for re-use. This style of work allows people to recycle what they discover about their data and their workflow with just a few extra keystrokes.

# Calculate reduced stats for data files at J = 100 c/bp.
for datafile in $*
do
    echo $datafile
    goostats -J 100 -r $datafile stats-$datafile
done

$ bash do-stats.sh *[AB].txt

$ bash do-stats.sh *[AB].txt | wc -l

# Calculate reduced stats for  A and Site B data files at J = 100 c/bp.
for datafile in *[AB].txt
do
    echo $datafile
    goostats -J 100 -r $datafile stats-$datafile
done

You can often guess someone's age by listening to how people talk about search (Figure 17). Just as young people use "Google" as a verb, crusty old Unix programmers use "grep". The word is a contraction of "global/regular expression/print", a common sequence of operations in early Unix text editors. It is also the name of a very useful command-line program.

grep finds and prints lines in files that match a pattern. For our examples, we will use a file that contains three haikus taken from a 1998 competition in Salon magazine:

The Tao that is seen
Is not the true Tao, until
You bring fresh toner.

With searching comes loss
and the presence of absence:
"My Thesis" not found.

Yesterday it worked
Today it is not working
Software is like that.

Let's find lines that contain the word "not":

$ grep not haiku.txt
Is not the true Tao, until
"My Thesis" not found
Today it is not working
$

Here, not is the pattern we're searching for. It's pretty simple: every alphanumeric character matches against itself. After the pattern comes the name or names of the files we're searching in. The output is the three lines in the file that contain the letters "not".

Let's try a different pattern: "day".

$ grep day haiku.txt
Yesterday it worked
Today it is not working
$

This time, the output is lines containing the words "Yesterday" and "Today", which both have the letters "day". If we give grep the -w flag, it restricts matches to word boundaries, so that only lines with the word "day" will be printed:

$ grep -w day haiku.txt
$

In this case, there aren't any, so grep's output is empty.

Another useful option is -n, which numbers the lines that match:

$ grep -n it haiku.txt
5:With searching comes loss
9:Yesterday it worked
10:Today it is not working
$

Here, we can see that lines 5, 9, and 10 contain the letters "it".

As with other Unix commands, we can combine flags. For example, since -i makes matching case-insensitive, and -v inverts the match, using them both only prints lines that don't match the pattern in any mix of upper and lower case:

$ grep -i -v the haiku.txt
You bring fresh toner.

With searching comes loss

Yesterday it worked
Today it is not working
Software is like that.
$

grep has lots of other options. To find out what they are, we can type man grep. man is the Unix "manual" command. It prints a description of a command and its options, and (if you're lucky) provides a few examples of how to use it:

$ man grep
GREP(1)                                                                                              GREP(1)

NAME
       grep, egrep, fgrep - print lines matching a pattern

SYNOPSIS
       grep [OPTIONS] PATTERN [FILE...]
       grep [OPTIONS] [-e PATTERN | -f FILE] [FILE...]

DESCRIPTION
       grep  searches the named input FILEs (or standard input if no files are named, or if a single hyphen-
       minus (-) is given as file name) for lines containing a match to the given PATTERN.  By default, grep
       prints the matching lines.
       …        …        …

OPTIONS
   Generic Program Information
       --help Print  a  usage  message  briefly summarizing these command-line options and the bug-reporting
              address, then exit.

       -V, --version
              Print the version number of grep to the standard output stream.  This version number should be
              included in all bug reports (see below).

   Matcher Selection
       -E, --extended-regexp
              Interpret  PATTERN  as  an  extended regular expression (ERE, see below).  (-E is specified by
              POSIX.)

       -F, --fixed-strings
              Interpret PATTERN as a list of fixed strings, separated by newlines, any of  which  is  to  be
              matched.  (-F is specified by POSIX.)
    …        …        …

$ grep -E '^.o' haiku.txt
You bring fresh toner.
Today it is not working
Software is like that.

While grep finds lines in files, the find command finds files themselves. Again, it has a lot of options; to show how the simplest ones work, we'll use the directory tree in Figure 18:

Vlad's home directory contains one file called notes.txt and four subdirectories: thesis (which is sadly empty), data (which contains two files first.txt and second.txt), a tools directory that contains the programs format and stats, and an empty subdirectory called old.

For our first command, let's run find . -type d. . is the directory where we want our search to start; -type d means "things that are directories". Sure enough, find's output is the names of the five directories in our little tree (including ., the current working directory):

$ find . -type d
./
./data
./thesis
./tools
./tools/old
$

If we change -type d to -type f, we get a listing of all the files instead:

$ find . -type f
./data/first.txt
./data/second.txt
./notes.txt
./tools/format
./tools/stats
$

find automatically goes into subdirectories, their subdirectories, and so on to find everything that matches the pattern we've given it. If we don't want it to, we can use -maxdepth to restrict the depth of search:

$ find . -maxdepth 1 -type f
./notes.txt
$

The opposite of -maxdepth is -mindepth, which tells find to only report things that are at or below a certain depth. -mindepth 2 therefore finds all the files that are two or more levels below us:

$ find . -mindepth 2 -type f
./data/first.txt
./data/second.txt
./tools/format
./tools/stats
$

Another option is -empty, which restricts matches to empty files and directories:

$ find . -empty
./thesis
./tools/old
$

Now let's try matching by name:

$ find . -name *.txt
./notes.txt
$

We expected it to find all the text files, but it only prints out ./notes.txt. The problem is that the shell expands wildcard characters like * before commands run. Since *.txt in the current directory expands to notes.txt, the command we actually ran was:

$ find . -name notes.txt

find did what we asked; we just asked for the wrong thing.

To get what we want, let's do what we did with grep: put *.txt in single quotes to prevent the shell from expanding the * wildcard. This way, find actually gets the pattern *.txt, not the expanded filename notes.txt:

$ find . -name '*.txt'
./data/first.txt
./data/second.txt
./notes.txt
$

As we said earlier, the command line's power lies in combining tools. We've seen how to do that with pipes; let's look at another technique. As we just saw, find . -name '*.txt' gives us a list of all text files in or below the current directory. How can we combine that with wc -l to count the lines in all those files?

One way is to put the find command inside $():

$ wc -l $(find . -name '*.txt')
  70  ./data/first.txt
 420  ./data/second.txt
  30  ./notes.txt
 520  total
$

When the shell executes this command, the first thing it does is run whatever is inside the $(). It then replaces the $() expression with that command's output. Since the output of find is the three filenames ./data/first.txt, ./data/second.txt, and ./notes.txt, the shell constructs the command:

$ wc -l ./data/first.txt ./data/second.txt ./notes.txt

which is what we wanted. This expansion is exactly what the shell does when it expands wildcards like * and ?, but lets us use any command we want as our own "wildcard".

It's very common to use find and grep together. The first finds files that match a pattern; the second looks for lines inside those files that match another pattern. Here, for example, we can find PDB files that contain iron atoms by looking for the string "FE" in all the .pdb files below the current directory:

$ grep FE $(find . -name '*.pdb')
./human/heme.pdb:ATOM  25  FE  1  -0.924  0.535  -0.518
$

# Compare all pairs of files.
for left in $*
do
    for right in $*
    do
        echo $left $right $(goodiff $left $right)
    done
done

$ bash pairwise.sh stats-*.txt

echo stats-NENE01729A.txt stats-NENE01729A.txt $(goodiff stats-NENE01729A.txt stats-NENE01729A.txt)
echo stats-NENE01729A.txt stats-NENE01729B.txt $(goodiff stats-NENE01729A.txt stats-NENE01729B.txt)
echo stats-NENE01729A.txt stats-NENE01736A.txt $(goodiff stats-NENE01729A.txt stats-NENE01736A.txt)
...

echo stats-NENE01729A.txt stats-NENE01729A.txt files are identical
echo stats-NENE01729A.txt stats-NENE01729B.txt 0.97182
echo stats-NENE01729A.txt stats-NENE01736A.txt 0.45223
...

stats-NENE01729A.txt stats-NENE01729A.txt files are identical
stats-NENE01729A.txt stats-NENE01729B.txt 0.97182
stats-NENE01729A.txt stats-NENE01736A.txt 0.45223
...

$ bash pairwise.sh stats-*.txt | grep -v 'files are identical'

for left in $*
do
    for right in $*
    do
        echo $left $right $(goodiff $left $right)
    done
done | grep -v 'files are identical'

(for left in $*
do
    for right in $*
    do
        echo $left $right $(goodiff $left $right)
    done
done) | grep -v 'files are identical'

(for left in $*
do
    for right in $*
    do
        echo $(goodiff $left $right) $left $right
    done
done) | grep -v 'files are identical'

(for left in $*
do
    for right in $*
    do
        echo $(goodiff $left $right) $left $right
    done
done) | grep -v 'files are identical' | sort -n -r | head -100

(for left in $*
do
    for right in $*
    do
        echo $(goodiff $left $right) $left $right
    done
done) \
| grep -v 'files are identical' \
| sort -n -r \
| head -100

$ bash pairwise.sh stats-*.txt > top100.txt