Shell redirection can be confusing. I’ve always wondered what 2>&1 means and why it’s needed, but as many others, I just copy/pasted things from the internet into my terminal and hoped for the best. What could possibly go wrong?

But no more! It’s finally time to look behind the curtain and decipher and understand what’s actually happening.

File Descriptors

To understand how redirection works, you first have to look at what the numbers actually represent: file descriptors (FD).

A file descriptor is a “process-unique identifier” or “handle” to a file or another I/O resource. They’re part of the POSIX API and each process “should” have three standard POSIX FDs:

Name            | A.K.A. | Number
----------------+--------+--------
Standard Input  | stdin  |   0
Standard Output | stdout |   1
Standard Error  | stderr |   2

You can actually check where these file descriptors are pointing at:

$ lsof +f g -ap $$ -d 0,1,2

# EXAMPLE OUTPUT FROM LINUX MACHINE:
# COMMAND    PID USER   FD   TYPE  FILE-FLAG DEVICE SIZE/OFF NODE NAME
# zsh     212233  ben    0u   CHR RW,0x80000  136,4      0t0    7 /dev/pts/4
# zsh     212233  ben    1u   CHR RW,0x80000  136,4      0t0    7 /dev/pts/4
# zsh     212233  ben    2u   CHR RW,0x80000  136,4      0t0    7 /dev/pts/4

Don’t worry about any warning that the command might output, as lsof checks all file systems and may lack the required permissions to do so. However, as you can see in the example output, the 3 FDs, 0, 1, and 2, are all mapped to a pts (Pseudoterminal). Depending on your terminal, it might be something different. On my Mac, the standard FDs point to /dev/ttys006.

For simplicity reasons, I’ll use /dev/tty0 as it usually represents the current virtual console and is found in many guides.

Writing to the Standard File Descriptors

Most programming languages have means to write to either output FDs, or read from the input, like in Java:

// STDIN
var reader = new InputStreamReader(System.in);

// STDOUT
System.out.println("write to stdout");

// STDERR
System.err.println("write to stderr");

… or Go:

// STDIN
reader := bufio.NewReader(os.Stdin)

// STDOUT
os.Stdout.WriteString("Write to stdout")
fmt.Fprintln(os.Stdout, "or use the fmt package")
fmt.Println("or write to stdout by default")

// STDERR
os.Stderr.WriteString("Write to stderr")
fmt.Fprintln(os.Stderr, "or use the fmt package")

Redirecting File Descriptors

The main reason you might want to redirect the standard FDs is how they’re used by their owning process, which might not match our requirements.

Maybe you want to save a command’s output into a file on disk, or read a file for input instead from stdin. Or you want to “connect” two commands, passing the first output to the second command’s input. These are all use cases for redirects.

By default, the standard FDs are mapped according to their direction to the same thing:

(0) STDIN  <─── /dev/tty0
(1) STDOUT ───> /dev/tty0
(2) STDERR ───> /dev/tty0

There are 3 operators for redirecting FDs:

• Output redirection with > (greater than)
• Input redirection with < (less than)
• Combining output and input with | (pipe)

Let’s take a look at all of them and their different use cases!

Output Redirection to a File

The simplest and maybe most common redirection is to point the stdout FD towards a file:

(0) STDIN  <─── /dev/tty0
(1) STDOUT ───> /location/to/file
(2) STDERR ───> /dev/tty0

To achieve this, you just need to combine the correct source, operator, and target:

Source:
1 (representing stdout)

Operator:
> (output redirection)

Target:
A file on disk

If you want to save the output from Gradle’s dependencies task into a file, this would look like this:

./gradlew dependencies 1> dependency-tree.txt

As stdout is, well, the “standard output”, this commonly used type of redirection is the default FD for the > operators, so you don’t need to add the FD at all, which makes it easier on the eyes and more straight-forward:

./gradlew dependencies > dependency-tree.txt

To redirect only error output, you need to redirect 2 (stderr) in the same way as before, so the redirection looks like this: 2> /location/to/file

The space between the operator and target is optional and often omitted, so the redirection builds a single block.

A great use case for redirecting stderr is to suppress the output completely by redirecting the magical black hole of any system: /dev/null

For example, the find command might output “Permission denied” errors, which you want to ignore. By redirecting the stderr FD to /dev/null, only the successful output will be displayed, or redirected to a file:

find . -name '*.txt' 2>/dev/null > all-txt-files.txt

Or, if you’re interested in which directories have the wrong permissions, you can easily create an error log file:

find . -name '*.txt' 2>error.log > all-txt-files.txt

Input Redirection from a File

Redirecting a file as input is equivalent to redirecting the output to one, expect using < instead.

The used operator actually matches the arrowhead on the direction:

(0) STDIN  <─── /location/to/file
(1) STDOUT ───> /dev/tty0
(2) STDERR ───> /dev/tty0

Let’s say you want to extract the database port from a JSON config file with jq. To do so, you can redirect the config file to jq:

jq '.connection.port' --raw-output < config.json

Most commands support a file as one of their arguments, so redirecting a file to stdin might seem superfluous. However, the possibility of redirection enables the next type of redirection: pipes.

Connection Commands with Pipes

The | (pipe) operator simply combines a command’s stdout with the stdin following the operator.

For example, looking up all markdown files and sorting the output can be done by using the output of find as input for sort:

find . -type f -name '*.md' | sort

Thinking in FDs, the following is happening:

(0) STDIN  <─── /dev/tty0   ╭──> (0) STDIN
(1) STDOUT ─────────────────╯    (1) STDOUT ───> /dev/tty0
(2) STDERR ───> /dev/tty0        (2) STDERR ───> /dev/tty0

This works because any redirections are connected before the commands are actually executed.

Even More Shenanigans with File Descriptors

We looked at the three fundamental operators and use cases, but there’s a lot more to FD redirection. And even more important, the common pitfalls and errors you may encounter.

Duplicating File Descriptors

As seen in the section about | (pipe), it’s an easy way to connect stdout of one process with stdin of another. But what about redirecting more than one FD? This article opened with the redirection 2>&1 so let’s dissect it and learn what it does.

find . -type f -name '*.md' 2>&1 | sort

The & (ampersand) modifier indicates that the left side of the operator “duplicates” the target of the right side. In this case, that means that stderr (2) redirects wherever stdout (1) is going:

(0) STDIN  <─── /dev/tty0  ╭─────> (0) STDIN
                           │╭───────^ 
(1) STDOUT ────────────────╯│      (1) STDOUT ───> /dev/tty0
(2) STDERR ─────────────────╯      (2) STDERR ───> /dev/tty0

Be aware that FDs are always duplicated, not aliased! The arrows in the graphs represent distinct FDs from the source to the target.

To simplify the call, you can use the shortcut |& to redirect both stdout and stderr into the next command and, therefore, removing the additional 2>&1.

Order Matters

As your requirements might be more complex than “redirect x to y”, you can use multiple redirection commands to set up the FDs as you need.

For example, you want to output both stdout and stderr to the same file. Using what we’ve learned so far, this would require two steps:

• Redirecting stderr to stdout (2>&1)
• Redirecting stdout to a file (> file)

The resulting call would most likely look like this:

command 2>&1 >file

Even though you’d think that’s the obvious solution, it’s a common error when dealing with multiple redirections. It’s time for another redirection dissection!

Redirections are set up from left to right. We start as usual:

(0) STDIN  <─── /dev/tty0
(1) STDOUT ───> /dev/tty0
(2) STDERR ───> /dev/tty0

Then, the 2>&1 gets set up and changes the FDs to this:

(0) STDIN  <─── /dev/tty0
(1) STDOUT ───> /dev/tty0
(2) STDERR ───> /dev/tty0

Well, nothing happened, but why? As stderr already has its distinct FD to the target of stdout, there’s no need to duplicate it.

The second redirect, >file, will lead to this:

(0) STDIN  <─── /dev/tty0
(1) STDOUT ───> file
(2) STDERR ───> /dev/tty0

There’s still a file with the contents of stdout, and you might think there are just no errors.

So the correct order for the two redirections is: command >file 2>&1

First, stdout is redirected to file:

(0) STDIN  <─── /dev/tty0
(1) STDOUT ───> file
(2) STDERR ───> /dev/tty0

Then, stderr duplicates the target of stdout:

(0) STDIN  <─── /dev/tty0
(1) STDOUT ───> file
(2) STDERR ───> file

As with the other common use cases, there’s a shortcut to simplify the redirections and don’t mix up the order:

command &>file

Which one to prefer is up to you, as there are multiple options available:

command &>file
command >file 2>&1
command 1>file 2>&1

Appending Vs. Overwrite files

The output redirection >file always overrides file if it exists. If you want to append to it instead, you can use >>.

In the case of a non-existing file, it gets created either way.

Preventing Accidental Overwrites

As > and < share the same key, mixing them up is easy, and accidentally destroying a file by redirecting from stdout to file instead of redirecting file to stdin. Or you typed a single > instead of the appending variant >>.

You can use the shell option set -o noclobber to prevent > from overwriting a pre-existing file.

If the need to override a file arises, you don’t have to disable and re-enable the option each time, you can simply force it with command >|file.

Reading and Writing the Same File

Another common error is trying to modify a file with a command that writes to stdout, like sed:

# THIS WON'T WORK
sed -u 's/fizz/buzz/' file >file

As redirections are set up before executing the command, stdout gets redirected to file which incidentally truncates file, as > is used.

In the case of sed, you can use the -i option for in-place editing. A more general solution is using | and the tee command, as it reads from stdin and writes to the provided file:

sed -u 's/fizz/buzz/' file | tee file

Conclusion

Shell redirections are powerful tools for building quite intricate command pipelines and bending the I/O to your will. That said, they’re also quite confusing if you’re not used to them. However, knowing the basics, like the different operators and FDs, will improve your shell productivity and help you to decipher all those weird one-liners you might copy into your shell.

If you want to know more about shell redirection, you can always consult the man page of bash directly in your terminal:

man -Len -Pless\ +/^REDIRECTION bash

Resources

• Bash Reference Manual (gnu.org)
• Zsh Redirection (zsh.sourceforge.io)
• How to manipulate files with shell redirection and pipelines in Linux (redhat.com)