Learn the Linux Command - uniq

Remove duplicates with ease with the 'uniq' Linux command

Businessman removing binder from shelf

Anthony Lee / OJO Images / Getty Images

The uniq command within the Linux shell prints the unique lines in a sorted file, retaining only one of a run of matching lines. Optionally, it can show only lines that appear exactly once, or lines that appear more than once. This program requires sorted input because it compares only consecutive lines—it won't sort the file for you.

Synopsis

The command takes the following general form:

uniq [-cdu] [-f skip-fields] [-s skip-chars] [-w check-chars] [-#skip-fields] [+#skip-chars] [--count] [--repeated] [--unique] [--skip-fields=skip-fields] [--skip-chars=skip-chars] [--check-hars=check-chars] [--help] [--version] [infile] [outfile]

Options

Use the following flags to fine-tune how the program executes:

  • -u, --unique: Only print unique lines.
  • -d, --repeated: Only print duplicate lines.
  • -c, --count: Print the number of times each line occurred along with the line.
  • -number, -f, --skip-fields=number: In this option, number is an integer representing the number of fields to skip over before checking for uniqueness. The first number fields, along with any blanks found before number fields is reached, are skipped over and not counted. Fields are defined as a string of non-space, non-tab characters, that are separated from each other by spaces and tabs.
  • +number, -s, --skip-chars=number: In this option, number is an integer representing the number of characters to skip over before checking for uniqueness. The first number characters, along with any blanks found before number characters is reached, are skipped over and not counted. If you use both the field and character skipping options, fields are skipped over first.
  • -w, --check-chars=number: Specify the number of characters to compare in the lines, after skipping any specified fields and characters. Normally the entire rest of the lines are compared.
  • --help: Print a usage message and exit with a status code indicating success.
  • --version: Print version information on standard output then exit.

Example

The command:

sort myfile | uniq

eliminates duplicate lines from the stream (the symbol "|" pipes the output from sort myfile to the uniq command).

Use the man command (% man) to see how a command is used on your particular computer.