uniq (Unique) - Text-Fu

The uniq command compares each input line with the preceding line. It can collapse, count, or select groups of adjacent equal lines, but it does not search the entire file for separated duplicates.

Collapsing Adjacent Duplicate Lines

Suppose reading.txt contains grouped values:

book
book
paper
paper
article
article
magazine

Run uniq with no filtering option to print one representative line from each adjacent group:

$ uniq reading.txt
book
paper
article
magazine

The input file remains unchanged because the result goes to stdout.

CheckWhat does uniq reading.txt do by default?

Counting Adjacent Groups

Use -c to prefix each output group with its number of consecutive input lines:

$ uniq -c reading.txt
      2 book
      2 paper
      2 article
      1 magazine

These are run lengths, not global totals unless all equal lines have first been made adjacent.

CheckWhat does the count from uniq -c represent?

Selecting Unique or Repeated Groups

Use -u to print only groups containing exactly one line:

$ uniq -u reading.txt
magazine

Use -d to print one representative line from each adjacent group containing more than one line:

$ uniq -d reading.txt
book
paper
article

GNU uniq -D prints every line from repeated groups, whereas lowercase -d prints each repeated group's value once.

CheckWhich command prints only adjacent groups that occur exactly once?

CheckWhich command prints one line for each adjacent group that appears more than once?

Grouping Separated Duplicates

If equal lines are separated, they form different groups:

book
paper
book
paper
article
magazine
article

Running uniq on this file will produce a surprising result:

$ uniq reading.txt
book
paper
book
paper
article
magazine
article

No lines are collapsed because neighboring values differ. Sort first when changing the order is acceptable and you want equal complete lines grouped together:

$ sort reading.txt | uniq
article
book
magazine
paper

Use a consistent locale and comparison policy across both steps. sort -u reading.txt can also sort and retain one line per equal sort key in a single command.

CheckEqual lines are scattered through reading.txt, and output order may change. Which pipeline produces one sorted copy of each distinct complete line?

uniq reads stdin when no input file is named, which is why it fits naturally after sort. GNU options such as -i can ignore case, while -f, -s, and -w can skip or limit comparison regions; use them only when equality should be defined by part of each line.

To practice grouping, counting, and filtering duplicates, try these hands-on labs:

Linux uniq Command: Duplicate Filtering - Learn how to use the Linux uniq command in combination with sort to identify, filter, and analyze duplicate lines in text files.
Linux sort Command: Text Sorting - Practice using the sort command to organize lines of text files, a crucial step before using uniq effectively.
Word Count and Sorting - Learn the essential Linux text processing tools wc (word count) and sort in this hands-on challenge. Learn to count lines, words, and characters, find frequent patterns, and sort data efficiently for various text analysis tasks.

Summary

You can now analyze adjacent groups of equal lines with uniq.

Collapse each adjacent duplicate group to one line.
Count consecutive occurrences with -c.
Select singleton groups with -u.
Select repeated groups with -d or GNU -D.
Sort first when separated duplicates must be grouped.

Practice

In the practice terminal below: 1. Make a grouped file: printf 'book\nbook\npaper\npaper\narticle\n' > /tmp/reading.txt 2. Collapse it: uniq /tmp/reading.txt 3. Count the groups: uniq -c /tmp/reading.txt Step 2 should print book, paper, article; step 3 should print 2 book, 2 paper, 1 article.

uniq collapses adjacent equal lines; -c adds a count per group.