I/O Redirection and Pipes Explained

I/O Redirection and Pipes Explained

Nearly every Linux command interacts with the world through three standard streams: input, output, and error. Understanding how to redirect and connect these streams is what turns a collection of individual commands into a pipeline capable of real work, and it underlies almost every non-trivial command you will see in documentation or scripts.

The three standard streams

Every process starts with three streams already open:

StreamFile descriptorPurpose
stdin0Where a program reads input from
stdout1Where a program writes its normal output
stderr2Where a program writes error and diagnostic messages

In an interactive terminal session, all three are connected to your terminal by default: stdin reads what you type, stdout and stderr both print to your screen. What redirection does is disconnect one or more of these from the terminal and connect it to something else instead, typically a file or another program.

Standard output and standard error are kept separate specifically so that error messages do not get mixed into a program’s normal output. This matters when you redirect stdout to a file expecting clean data, error messages would corrupt that data if the two streams were not separate.

Redirecting output with > and >>

ls > filelist.txt          # write output to filelist.txt, overwriting it
ls >> filelist.txt         # append output to filelist.txt, keeping existing content

> truncates the target file first, so any existing content is lost. >> appends to the end, leaving existing content intact. Using > when you meant >> is one of the most common ways people accidentally destroy data on the command line, since there is no confirmation prompt.

echo "First line" > log.txt
echo "Second line" >> log.txt
cat log.txt
# First line
# Second line

Redirecting input with <

< feeds a file’s contents into a command’s standard input, as if you had typed it:

sort < unsorted.txt
mysql -u root -p database_name < backup.sql

Many commands accept a filename directly as an argument instead (sort unsorted.txt), which does the same thing for that command specifically, but < works generically for any command that reads from stdin, including ones that only accept input that way.

Redirecting standard error

Standard error uses file descriptor 2 explicitly:

command 2> errors.txt          # send only stderr to errors.txt
command > output.txt 2> errors.txt   # stdout and stderr to separate files

To send both stdout and stderr to the same file, redirect stdout first, then point stderr at wherever stdout is currently going, using 2>&1:

command > combined.txt 2>&1

The order matters here. 2>&1 > combined.txt does not produce the same result, because at the point 2>&1 runs, stdout is still pointing at the terminal, so stderr gets duplicated to the terminal too, and only stdout ends up in the file afterward. Bash also provides a shorthand for the common case of combining both streams into one file:

command &> combined.txt

Discarding output with /dev/null

/dev/null is a special device file that silently discards anything written to it and returns end-of-file immediately when read from. It is the standard way to suppress output you do not want to see or keep:

command > /dev/null            # discard normal output, keep error messages visible
command 2> /dev/null           # discard error messages, keep normal output visible
command > /dev/null 2>&1       # discard everything

This pattern shows up constantly in cron jobs and scripts, where a command’s routine output is not useful and would otherwise clutter logs or trigger unwanted cron email notifications.

Pipes

A pipe (|) connects the standard output of one command directly to the standard input of the next, without an intermediate file:

command1 | command2

This is the mechanism that makes small, single-purpose Unix tools composable into pipelines that do far more than any one tool alone:

ps aux | grep firefox
history | grep ssh
cat access.log | grep 404 | wc -l

Pipelines can chain as many commands as needed:

cat access.log | grep 404 | awk '{print $1}' | sort | uniq -c | sort -rn

This example reads a log file, filters for 404 responses, extracts the IP address field, sorts them, counts occurrences of each unique IP, then sorts by count descending, a realistic example of stacking simple tools to answer a specific question (“which IPs generated the most 404 errors”) that no single tool answers directly.

xargs: turning piped input into arguments

Pipes connect stdout to stdin, but many commands, rm, cp, mv, and others, expect their targets as command-line arguments, not as text on standard input. This is where xargs comes in: it reads items from standard input and converts them into arguments for another command.

find . -name "*.tmp" | xargs rm

find prints matching filenames to stdout, and xargs takes that list and builds an rm command with each filename as an argument, effectively running rm file1.tmp file2.tmp file3.tmp. Without xargs, piping directly into rm (find . -name "*.tmp" | rm) does not work, because rm does not read filenames from stdin at all.

# Safer version: print what would be deleted first
find . -name "*.tmp" | xargs echo

# Handle filenames with spaces safely
find . -name "*.tmp" -print0 | xargs -0 rm

The -print0 and -0 combination separates filenames with a null character instead of whitespace, avoiding bugs when filenames contain spaces, which is a subtle but important detail for scripts that need to handle arbitrary filenames correctly.

Combining redirection and pipes

Redirection and pipes combine freely in real commands:

grep -r "TODO" . 2>/dev/null | sort | tee todos.txt

This searches recursively for “TODO”, discards permission-denied errors, sorts the results, and tee both prints the output to the terminal and saves it to todos.txt simultaneously. tee is specifically useful when you want to both see output live and capture it, rather than choosing one or the other with a plain redirect.

Frequently Asked Questions

What is the difference between > and >> in Linux?

Both redirect a command’s standard output to a file instead of the terminal, but > overwrites the file completely, replacing any existing content, while >> appends to the end of the file, preserving whatever was already there. Using > on a file you meant to append to is a common way to accidentally lose data, so when in doubt about whether a file already has content you want to keep, use >> or check the file first.

What are the three standard streams in Linux?

Every process has three standard streams by default: standard input (stdin, file descriptor 0), where a program reads input from; standard output (stdout, file descriptor 1), where a program writes its normal output; and standard error (stderr, file descriptor 2), where a program writes error and diagnostic messages, kept separate from normal output specifically so error messages are not mixed in with data by default. All three are connected to the terminal in an interactive session, but any of them can be redirected to a file, another program, or discarded.

How do I redirect both stdout and stderr to the same file?

Use command > file.txt 2>&1, in that order. This first redirects stdout to file.txt, then redirects stderr (file descriptor 2) to wherever stdout is currently pointing (file descriptor 1, via the 2>&1 syntax), which is now the file. The order matters: 2>&1 > file.txt does something different, since it duplicates stderr to the terminal (where stdout was still pointing at that moment) before stdout gets redirected to the file, leaving stderr going to the terminal instead of the file. Bash also supports a shorter form for the common case: command &> file.txt sends both streams to the file directly.

What does | (pipe) do in the shell?

A pipe connects the standard output of one command directly to the standard input of the next command, without creating an intermediate file. command1 | command2 means command2 receives whatever command1 writes to its standard output, as its own standard input. This lets you chain simple, single-purpose commands into a pipeline that performs a more complex task, such as ps aux | grep firefox | awk ‘{print $2}’, which lists processes, filters for a name, then extracts a column, each command doing one job.

What is xargs used for?

xargs takes input from standard input (typically piped in from another command) and converts it into arguments for a new command, rather than input on that command’s stdin. This matters because many commands, like rm, cp, and mv, expect their targets as command-line arguments, not as piped input; find . -name “.tmp” | rm does not work as expected because rm does not read filenames from stdin, but find . -name “.tmp” | xargs rm does, because xargs converts the piped filenames into arguments for rm.

How do I discard command output entirely?

Redirect it to /dev/null, a special device file that silently discards anything written to it: command > /dev/null discards standard output, command 2> /dev/null discards standard error, and command > /dev/null 2>&1 discards both. This is commonly used in scripts and cron jobs to suppress expected, non-actionable output while still allowing the script’s own explicit output or exit code to be checked.