linux uniq命令

The Linux uniq command whips through your text files looking for unique or duplicate lines. In this guide, we cover its versatility and features, as well as how you can make the most of this nifty utility.

Linux uniq命令在您的文本文件中快速查找唯一或重复的行。 在本指南中，我们介绍了它的多功能性和功能，以及如何充分利用这个漂亮的实用程序。

在Linux上查找文本的匹配行 (Finding Matching Lines of Text on Linux)

The uniq command is fast, flexible, and great at what it does. However, like many Linux commands, it has a few quirks—which is fine, as long as you know about them. If you take the plunge without a bit of insider know-how, you could well be left scratching your head at the results. We’ll point out these quirks as we go.

uniq命令快速，灵活，并且功能强大。 但是，就像许多Linux命令一样，它也有一些怪癖-只要您了解它们就可以了。 如果您在没有任何内幕专业知识的情况下进行尝试，那么您很可能会为结果感到挠头。 我们将指出这些怪癖。

The uniq command is perfect for those in the single-minded, designed-to-do-one-thing-and-do-it-well camp. That’s why it’s also particularly well-suited to work with pipes and play its part in command pipelines. One of its most frequent collaborators is sort because uniq has to have sorted input on which to work.

uniq命令非常适合那些专一，一劳永逸的人。 这就是为什么它也特别适合与管道一起工作并在命令管道中发挥作用的原因。 它的一个的最频繁的合作者是sort因为uniq必须有排序的文件在其上工作。

Let’s fire it up!

让我们点火！

不带选项运行uniq (Running uniq with No Options)

We’ve got a text file that contains the lyrics to Robert Johnson’s song I Believe I’ll Dust My Broom. Let’s see what uniq makes of it.

我们有一个文本文件，其中包含罗伯特·约翰逊(Robert Johnson)的歌曲《我相信我会除尘我的扫帚》的歌词。 让我们看看uniq是如何构成的。

We’ll type the following to pipe the output into less:

我们将键入以下内容以将输出传递到less ：

uniq dust-my-broom.txt | less

We get the entire song, including duplicate lines, in less:

我们得到的整首歌曲，包括重复的线条， less ：

That doesn’t seem to be either the unique lines nor the duplicate lines.

似乎既不是唯一的行也不是重复的行。

Right—because this is the first quirk. If you run uniq with no options, it behaves as though you used the -u (unique lines) option. This tells uniq to print only the unique lines from the file. The reason you see duplicate lines is because, for uniq to consider a line a duplicate, it must be adjacent to its duplicate, which is where sort comes in.

正确-因为这是第一个怪癖。 如果运行不带任何选项的uniq ，它的行为就好像您使用了-u (唯一行)选项一样。 这告诉uniq仅打印文件中的唯一行。 之所以会看到重复的行，是因为，为了使uniq认为行是重复的，它必须与它的重复相邻，这就是sort来源。

When we sort the file, it groups the duplicate lines, and uniq treats them as duplicates. We’ll use sort on the file, pipe the sorted output into uniq, and then pipe the final output into less.

当我们对文件进行排序时，它会将重复的行进行分组，并且uniq将它们视为重复的行。 我们将对文件使用sort ，将排序后的输出uniq给uniq ，然后将最终输出uniq给less 。

To do so, we type the following:

为此，我们键入以下内容：

sort dust-my-broom.txt | uniq | less

A sorted list of lines appears in less.

行的排序列表出现在less 。

The line, “I believe I’ll dust my broom,” definitely appears in the song more than once. In fact, it’s repeated twice within the first four lines of the song.

这首歌“我相信我会把扫帚除尘”肯定在歌曲中多次出现。 实际上，它在歌曲的前四行中重复了两次。

So, why is it showing up in a list of unique lines? Because the first time a line appears in the file, it’s unique; only the subsequent entries are duplicates. You can think of it as listing the first occurrence of each unique line.

那么，为什么它显示在唯一行列表中？ 因为第一次在文件中出现一行，所以它是唯一的。 仅后续条目重复。 您可以将其视为列出每个唯一行的第一个匹配项。

Let’s use sort again and redirect the output into a new file. This way, we don’t have to use sort in every command.

让我们再次使用sort并将输出重定向到一个新文件。 这样，我们不必在每个命令中都使用sort 。

We type the following command:

我们输入以下命令：

sort dust-my-broom.txt > sorted.txt

Now, we have a presorted file to work with.

现在，我们有了一个预分类的文件可以使用。

计数重复 (Counting Duplicates)

You can use the -c (count) option to print the number of times each line appears in a file.

您可以使用-c (计数)选项来打印每行出现在文件中的次数。

Type the following command:

键入以下命令：

uniq -c sorted.txt | less

Each line begins with the number of times that line appears in the file. However, you’ll notice the first line is blank. This tells you there are five blank lines in the file.

每行以该行出现在文件中的次数开头。 但是，您会注意到第一行是空白。 这告诉您文件中有五个空白行。

If you want the output sorted in numerical order, you can feed the output from uniq into sort. In our example, we’ll use the -r (reverse) and -n (numeric sort) options, and pipe the results into less.

如果要按数字顺序对输出进行排序，可以将uniq的输出输入sort 。 在我们的示例中，我们将使用-r (反向)和-n (数字排序)选项，并将结果传递给less 。

We type the following:

我们输入以下内容：

uniq -c sorted.txt | sort -rn | less

The list is sorted in descending order based on the frequency of each line’s appearance.

该列表根据每行出现的频率以降序排列。

只列出重复的行 (Listing Only Duplicate Lines)

If you want to see only the lines that are repeated in a file, you can use the -d (repeated) option. No matter how many times a line is duplicated in a file, it’s listed only once.

如果只想查看文件中重复的行，则可以使用-d (重复)选项。 无论文件中一行被重复多少次，它只会被列出一次。

To use this option, we type the following:

要使用此选项，我们输入以下内容：

uniq -d sorted.txt

The duplicated lines are listed for us. You’ll notice the blank line at the top, which means the file contains duplicate blank lines—it isn’t a space left by uniq to cosmetically offset the listing.

为我们列出了重复的行。 您会注意到顶部的空白行，这意味着该文件包含重复的空白行-这不是uniq用来修饰列表的空白。

We can also combine the -d (repeated) and -c (count) options and pipe the output through sort. This gives us a sorted list of the lines that appear at least twice.

我们还可以结合使用-d (重复)和-c (计数)选项，并通过sort传递输出。 这给了我们至少出现两次的行的排序列表。

Type the following to use this option:

输入以下内容以使用此选项：

uniq -d -c sorted.txt | sort -rn

列出所有重复的行 (Listing All Duplicated Lines)

If you want to see a list of every duplicated line, as well as an entry for each time a line appears in the file, you can use the -D (all duplicate lines) option.

如果要查看每个重复行的列表，以及每次在文件中出现一行的条目，则可以使用-D (所有重复行)选项。

To use this option, you type the following:

要使用此选项，请键入以下内容：

uniq -D sorted.txt | less

The listing contains an entry for each duplicated line.

该清单包含每个重复行的条目。

If you use the --group option, it prints every duplicated line with a blank line either before (prepend) or after each group (append), or both before and after (both) each group.

如果使用--group选项，它将在每个组之前( prepend )或之后( append )，或在每个组之前和之后( both )都在每行重复的行上打印空白行。

We’re using append as our modifier, so we type the following:

我们使用append作为修饰符，因此我们输入以下内容：

uniq --group=append sorted.txt | less

The groups are separated by blank lines to make them easier to read.

这些组用空行分隔，以使它们更易于阅读。

检查一定数量的字符 (Checking a Certain Number of Characters)

By default, uniq checks the entire length of each line. If you want to restrict the checks to a certain number of characters, however, you can use the -w (check chars) option.

默认情况下， uniq检查每行的整个长度。 但是，如果要将检查限制为一定数量的字符，则可以使用-w (检查字符)选项。

In this example, we’ll repeat the last command, but limit the comparisons to the first three characters. To do so, we type the following command:

在此示例中，我们将重复最后一个命令，但将比较限制为前三个字符。 为此，我们键入以下命令：

uniq -w 3 --group=append sorted.txt | less

The results and groupings we receive are quite different.

我们收到的结果和分组是完全不同的。

All lines that start with “I b” are grouped together because those portions of the lines are identical, so they’re considered to be duplicates.

所有以“ I b”开头的行都被分组在一起，因为这些行的那些部分是相同的，因此它们被认为是重复的。

Likewise, all lines that start with “I’m” are treated as duplicates, even if the rest of the text is different.

同样，以“ I'm”开头的所有行都被视为重复行，即使文本的其余部分不同。

忽略一定数量的字符 (Ignoring a Certain Number of Characters)

There are some cases in which it might be beneficial to skip a certain number of characters at the beginning of each line, such as when lines in a file are numbered. Or, say you need uniq to jump over a timestamp and start checking the lines from character six instead of from the first character.

在某些情况下，在每一行的开头略过一定数量的字符可能是有益的，例如对文件中的行进行编号时。 或者，说您需要uniq跳过时间戳并开始检查从字符6而不是从第一个字符开始的行。

Below is a version of our sorted file with numbered lines.

下面是带有编号行的已排序文件的版本。

If we want uniq to start its comparison checks at character three, we can use the -s (skip chars) option by typing the following:

如果我们希望uniq在第三个字符处开始比较检查，则可以通过键入以下命令使用-s (跳过字符)选项：

uniq -s 3 -d -c numbered.txt

The lines are detected as duplicates and counted correctly. Notice the line numbers displayed are those of the first occurrence of each duplicate.

将这些行检测为重复行并正确计数。 注意，显示的行号是每个重复项第一次出现的行号。

You can also skip fields (a run of characters and some white space) instead of characters. We’ll use the -f (fields) option to tell uniq which fields to ignore.

您也可以跳过字段(一系列字符和一些空白)来代替字符。 我们将使用-f (字段)选项来告诉uniq要忽略哪些字段。

We type the following to tell uniq to ignore the first field:

我们输入以下内容告诉uniq忽略第一个字段：

uniq -f 1 -d -c  numbered.txt

We get the same results we did when we told uniq to skip three characters at the start of each line.

当我们告诉uniq在每一行的开头跳过三个字符时，我们得到的结果相同。

忽略大小写 (Ignoring Case)

By default, uniq is case-sensitive. If the same letter appears capped and in lowercase, uniq considers the lines to be different.

默认情况下， uniq区分大小写。 如果相同的字母显示为大写且小写，则uniq认为行是不同的。

For example, check out the output from the following command:

例如，检查以下命令的输出：

uniq -d -c sorted.txt | sort -rn

The lines “I Believe I’ll dust my broom” and “I believe I’ll dust my broom” aren’t treated as duplicates because of the difference in case on the “B” in “believe.”

由于“相信”中“ B”的大小写不同，因此“我相信我会扫帚”和“我相信我会扫帚”这行不会被重复。

If we include the -i (ignore case) option, though, these lines will be treated as duplicates. We type the following:

但是，如果我们包括-i (忽略大小写)选项，则这些行将被视为重复行。 我们输入以下内容：

uniq -d -c -i sorted.txt | sort -rn

The lines are now treated as duplicates and grouped together.

现在，这些行被视为重复行并分组在一起。

---

---

Linux puts a multitude of special utilities at your disposal. Like many of them, uniq isn’t a tool you’ll use every day.

Linux提供了许多特殊实用程序供您使用。 像其中许多工具一样， uniq并不是您每天都会使用的工具。

That’s why a big part of becoming proficient in Linux is remembering which tool will solve your current problem, and where you can find it again. If you practice, though, you’ll be well on your way.

这就是为什么精通Linux的很大一部分原因是要记住哪种工具可以解决您当前的问题，以及在哪里可以找到它。 但是，如果您练习的话，将会很顺利。

Or, you can always just search How-To Geek—we probably have an article on it.

或者，您始终可以只搜索How-To Geek-我们可能在上面有一篇文章。

翻译自: https://www.howtogeek.com/533406/how-to-use-the-uniq-command-on-linux/

linux uniq命令