pipe and tee

Pipelines connect small commands so data can flow between them without an intermediate file. The tee command can copy part of that flow to a file while continuing to send it onward.

Connecting Commands with |

Suppose a directory listing is too long to read at once:

$ ls -la /etc

Place the pipe operator, |, between commands to connect the left command's stdout to the right command's stdin:

$ ls -la /etc | less

The shell starts the pipeline commands and arranges the stream connection. The commands can work concurrently: less can begin reading before ls has produced its entire listing.

Keeping stderr Separate

A plain | carries stdout only. Stderr from the left command keeps its previous destination, which is often the terminal:

$ find /etc -name "*.conf" | less

Matching pathnames go through the pipe, while permission diagnostics can still appear directly on the terminal. Redirect stderr separately when you need different behavior:

$ find /etc -name "*.conf" 2> find-errors.log | less

Copying a Stream with tee

tee reads stdin, writes a copy to each named file, and also writes the same data to stdout:

$ ls | tee listing.txt

Here, listing.txt receives the listing and tee's stdout remains connected to the terminal. By default, tee creates or truncates the named file, just like >.

Use -a when the file should be appended instead of replaced:

$ date | tee -a activity.log

Saving an Intermediate Result

Put tee in the middle of a pipeline to save an intermediate stream and continue processing it:

$ ls -la /etc | tee etc-listing.txt | grep "conf"

This pipeline:

1. Produces the complete long listing.
2. Saves that complete stream in etc-listing.txt.
3. Sends the same stream to grep, which prints only lines containing conf.

The file contains the data before grep filters it. If you want only the filtered lines in the file, put tee after grep.

To practice pipelines and stream copying, try these hands-on labs:

1. Redirecting Input and Output in Linux - Practice controlling data flow from commands by manipulating standard output (stdout), standard error (stderr), and standard input (stdin) using operators like >, >>, 2>, and the tee command.
2. Sequence Control and Pipeline - Learn to control command execution sequences, utilize pipelines, and leverage powerful text processing tools like cut, grep, wc, sort, and uniq.
3. Data Stream Redirection - Learn the art of Linux stream redirection, including manipulating standard input, output, and error streams, combining outputs, and utilizing /dev/null.

Summary

You can now connect commands and preserve selected points in a data stream.

1. Pipe stdout from one command into another command's stdin.
2. Redirect stderr separately when required.
3. Copy input to both a file and stdout with tee.
4. Append with tee -a instead of replacing a file.
5. Position tee before or after a filter deliberately.