by Jimmy Zhang

吉米·张(Jimmy Zhang)

I once spent a couple of frustrating days at work learning how to properly deal with Unicode strings in Python. During those two days, I ate a lot of snacks — roughly one bag of goldfish per one of these errors encountered, which should be all too familiar to those who program with Python:

我曾经花了几天令人沮丧的工作来学习如何正确处理Python中的Unicode字符串。 在这两天中，我吃了很多零食-遇到这些错误之一，大约要吃一袋金鱼，这对于使用Python进行编程的人来说应该太熟悉了：

UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xf0 in position 0: ordinal not in range(128)

While solving my issue, I did a lot of googling, which pointed me to a few indispensable articles. But as great as they are, they were all written without the help of a crucial aspect of communication in today’s day and age.

在解决我的问题时，我进行了大量的谷歌搜索，这为我指出了 一些 必不可少的 文章 。 但是，尽管它们是如此出色，但它们的编写都是在当今时代没有交流的关键方面的帮助下进行的。

That is: they were all written without the help of emoji.

也就是说：它们都是在没有表情符号帮助的情况下编写的。

So, in order to take advantage of this situation, I decided to write my own guide to understanding Unicode, with plenty of faces and icons rendered along the way ?✌?.

因此，为了利用这种情况，我决定编写自己的理解Unicode的指南，并在方向上绘制了大量的面Kong和图标。

Before diving into technical details, let’s begin with a fun question. What is your favorite emoji?

在深入探讨技术细节之前，让我们从一个有趣的问题开始。 你最喜欢的表情符号是什么？

Mine is the “face with open mouth”, which looks like this ?— with one major caveat. What you see is actually dependent on the platform you are using to read this post!

我的是“ 张开嘴巴的脸 ”，看起来像这样吗？ 您所看到的实际上取决于您用于阅读本文的平台！

Viewed on my Mac, the emoji looks like a yellow bowling ball. On my Samsung tablet, the eyes are black and circular, accentuated by a white dot which betrays a greater depth of emotion.

在Mac上查看，表情符号看起来像一个黄色的保龄球。 在我的三星平板电脑上，眼睛是黑色和圆形的，并带有白色的点，突出了更深层次的情感。

Copy and paste the emoji (?) into Twitter, and you’ll see something completely different. Copy and paste it into messenger.com, however, and you’ll see why it is my favorite.

将表情符号(？)复制并粘贴到Twitter中，您将看到完全不同的东西。 但是，将其复制并粘贴到messenger.com中，您将明白为什么它是我的最爱。

???? Why are they all different?

???? 他们为什么都不同？

Note: As of July 9th, 2018: Messenger seems to have updated their emoji icons, so the icon at the top right no longer applies. ?

注意：截至2018年7月9日：Messenger似乎已更新了其表情符号图标，因此右上角的图标不再适用。 ？

This fun little mystery is our segue into the world of Unicode, as emojis have been part of the Unicode Standard since 2010. Aside from giving us emoji, Unicode is important because it is the Internet’s preferred choice for the consistent “encoding, representation, and handling of text”.

这个有趣的小谜团是我们进入Unicode世界的话题，因为表情符号自2010年以来就已成为Unicode标准的一部分。除了为我们提供表情符号外，Unicode十分重要，因为它是Internet上一致的“ 编码，表示法和处理文字 ”。

Unicode和编码：简要入门 (Unicode & Encoding: A Brief Primer)

As with many topics, the best way to understand Unicode is to know the context surrounding its creation — and for that, Joel Spolsky’s article is required reading.

与许多主题一样，理解Unicode的最佳方法是了解其创建的上下文-为此，需要阅读Joel Spolsky的文章 。

代码点 (Code Points)

Since we’ve now entered the world of Unicode, we need to first dissociate emojis from the wonderfully expressive icons they are, and associate them with something much less exciting. So instead of thinking about emojis in terms of the things or the emotions that they represent, we will instead think about each emoji as a plain number. This number is known as a code point.

由于我们现在已经进入Unicode世界，因此我们需要首先将表情符号与它们具有奇妙表现力的图标分离，然后将它们与不那么令人兴奋的东西相关联。 因此，我们不用按照表情或表情所代表的情感来考虑表情符号，而是将每个表情符号视为一个简单的数字。 此数字称为代码点 。

Code points are the key concept of Unicode, which was “designed to support the worldwide interchange, processing, and display of the written texts of the diverse languages…of the modern world.” It does so by associating virtually every printable character with an unique code point. Together, these characters comprise the Unicode character set.

代码点是Unicode的关键概念 ，它“旨在支持在世界范围内交换，处理和显示现代世界各种语言的书面文本”。 它通过将几乎每个可打印字符与唯一的代码点相关联来实现。 这些字符共同构成了Unicode 字符集 。

Code points are typically written in hexadecimal and prefixed with U+ to denote the connection to Unicode, representing characters from:

代码点通常以十六进制编写，并以U+前缀表示与Unicode的连接，表示来自以下方面的字符：

• exotic languages such as Telugu [ఋ | code point: U+0C0B]

  外来语言，例如泰卢固语 [ఋ| 代码点：U + 0C0B]

• chess symbols [♖ | code point: U+2656]

  象棋符号 [♖| 码点：U + 2656]

• and, of course, emojis [? | code point: U+1F64C]

  当然还有表情符号 [？ | 码点：U + 1F64C]

字形就是您所看到的 (Glyphs Are What You See)

The actual on-screen representation of code points are called glyphs, (the complete mapping of code points to glyphs is known as a font).

屏幕上代码点的实际表示形式称为字形 ( 完整的映射 指向字形的代码点称为字体 。

As an example, take this letter A, which is code point U+0041 in Unicode. The “A” you see with your eyes is a glyph — it looks like the way it does because it is rendered with Medium’s font. If you were to change the font to, Times New Roman for example, only the glyph of “A” would change — the underlying code point would not.

例如，以字母A为例，它是Unicode中的代码点U+0041 。 您用眼睛看到的“ A”是一个字形-看起来像它的方式，因为它是用Medium的字体呈现的。 如果将字体更改为Times New Roman，例如，只有字形“ A”会更改，而底层代码点则不会更改。

Glyphs are the answer to our little rendering mystery. Under the hood, all variations of the face with open mouth emoji point to the same code point, U+1F62E, but the glyph representing it varies by platform ?.

字形是我们小的渲染之谜的答案。 在引擎盖下，带有张开表情符号表情的脸部的所有变体都指向相同的代码点U+1F62E ，但表示它的标志符号随平台varies而变化。

代码点是抽象 (Code Points are Abstractions)

Because they say nothing about how they are rendered visually (requiring a font and a glyph to “bring them to life”), code points are said to be an abstraction.

因为它们没有说明它们在视觉上的呈现方式(需要字体和标志符号才能“使它们栩栩如生”)，所以代码点被认为是一种抽象。

But just as code points are an abstraction to end users, they are also abstractions to computers. This is because code points require a character encoding to convert them into the one thing which computers can interpret: bytes. Once converted to bytes, code points can be saved to files or sent over the network to another computer ?➡️?.

但是就像代码点是对最终用户的抽象一样，它们也是计算机的抽象。 这是因为代码点需要字符编码才能将其转换为计算机可以解释的一件事：字节。 一旦转换为字节，代码点就可以保存到文件或通过网络发送到另一台计算机。

UTF-8 is currently the world’s most popular character encoding. UTF-8 uses a set of rules to convert a code point into an unique sequence of (1 to 4) bytes, and vice versa. Code points are said to be encoded into a sequence of bytes, and sequences of bytes are decoded into code points. This Stack Overflow post explains how the UTF-8 encoding algorithm works.

UTF-8是目前世界上最流行的字符编码 。 UTF-8使用一组规则将代码点转换为唯一的字节序列(1到4个字节)，反之亦然。 据说代码点被编码为字节序列，并且字节序列被解码为代码点。 这篇Stack Overflow帖子介绍了UTF-8编码算法的工作原理。

However, even though UTF-8 is the predominant character encoding in the world, it is far from the only one. For example, UTF-16 is an alternative character encoding of the Unicode character set. The image below compares the UTF-8 and UTF-16 encodings of our emoji ?.

但是，即使UTF-8是世界上主要的字符编码，也远非唯一。 例如，UTF-16是Unicode字符集的替代字符编码。 下图比较了我们的表情符号？的UTF-8和UTF-16编码。

Problems arise when one computer encodes code points into bytes with one encoding, and another computer (or another process on the same computer) decodes those bytes with another.

当一台计算机使用一种编码将代码点编码为字节，而另一台计算机(或同一台计算机上的另一个进程)使用另一种编码对这些字节进行解码时，就会出现问题。

Luckily, UTF-8 is ubiquitous enough that, for the most part, we don’t have to worry about mismatched character encodings. But when they do occur, a familiarity with the concepts mentioned above is required to extricate yourself from the mess.

幸运的是，UTF-8普遍存在，因此在大多数情况下，我们不必担心字符编码不匹配。 但是当它们确实发生时，需要熟悉上述概念才能使自己摆脱混乱。

简要回顾 (Brief Recap)

• Unicode is a collection of code points, which are plain numbers typically written in hexadecimal and prefixed with U+. These code points map to virtually every printable character from the written languages around the world.

  Unicode是代码点的集合，这些代码点是通常以十六进制形式编写且以U+为前缀的纯数字。 这些代码点实际上映射到来自世界各地书面语言的每个可打印字符。

• Glyphs are the physical manifestation of a character. This guy ? is a glyph. A font is a mapping of code points to glyphs.

  字形是角色的物理表现。 这家伙 ？ 是一个字形。 A F ONT是码点到字形映射。

• In order to send them across the network or save them in a file, characters and their underlying code points must be encoded into bytes. A character encoding contains the details of how a code point is embedded into a sequence of bytes.

  为了通过网络发送它们或将它们保存在文件中，必须将字符及其基础代码点编码为字节。 字符编码包含有关如何将代码点嵌入到字节序列中的详细信息。

• UTF-8 is currently the world’s must popular character encoding. Given a code point, UTF-8 encodes it into a sequence of bytes. Given a sequence of bytes, UTF-8 decodes it into a code point.

  UTF-8当前是世界上必须流行的字符编码。 给定一个代码点，UTF-8会将其编码为字节序列。 给定字节序列，UTF-8 会将其解码为代码点。

一个实际的例子 (A Practical Example)

The correct rendering of Unicode characters involves traversing a chain, ranging from bytes to code points to glyphs.

正确呈现Unicode字符涉及遍历一条链，范围从字节到代码点再到字形。

Let’s now use a text editor to see a practical example of this chain — as well as the types of issues that can arise when things go awry. Text editors are perfect, because they involve all three parts of the rendering chain shown above.

现在，让我们使用文本编辑器查看该链的实际示例-以及出现问题时可能出现的问题类型。 文本编辑器是完美的，因为它们涉及上面显示的渲染链的所有三个部分。

Note: The following example was done on my MacOS using Sublime Text 3. And to give credit where credit is due: the beginning of this example is heavily inspired by this post from Philip Guo, which introduced me to the hexdump command (and a whole lot more).

注意：以下示例是在MacOS上使用Sublime Text 3完成的。并在应归功的地方致谢：该示例的开始很大程度上受到Philip Guo的这篇帖子的启发，他向我介绍了hexdump命令(以及整个更多)。

We’ll start with a text file containing a single character — my favorite “face with open mouth” emoji. For those who want to follow along, I’ve hosted this file in a Github gist, which you get locally with curl.

我们将从一个包含单个字符的文本文件开始-我最喜欢的“张开嘴的脸”表情符号。 对于那些想要继续学习的人，我将这个文件托管在Github gist中 ，您可以在本地使用curl 。

curl https://gist.githubusercontent.com/jzhang621/d7d9eb167f25084420049cb47510c971/raw/e35f9669785d83db864f9d6b21faf03d9e51608d/emoji.txt > emoji.txt

As we learned, in order for it be saved to a file, the emoji was encoded into bytes using a character encoding. This particular file was encoded using UTF-8, and we can use the hexdump command to examine the actual byte contents of the file.

据了解，为了将其保存到文件中，表情符号使用字符编码被编码为字节。 这个特定的文件是使用UTF-8编码的，我们可以使用hexdump命令检查文件的实际字节内容。

j|encoding: hexdump emoji.txt0000000 f0 9f 98 ae 0000004

The output of hexdump tells us the file contains 4 bytes total, each of which is written in hexadecimal. The actual byte sequence f0 9f 98 ae matches the expected UTF-8 encoded byte sequence, as shown below.

hexdump的输出告诉我们该文件总共包含4个字节，每个字节hexdump十六进制表示。 实际字节序列f0 9f 98 ae与预期的UTF-8编码字节序列匹配，如下所示。

Now, let’s open our file in Sublime Text, where we should see our single ? character. Since we see the expected glyph, we can assume Sublime Text used the correct character encoding to decode those bytes into code points. Let’s confirm by opening up the console View -> Show Console, and inspecting the view object that Sublime Text exposes as part of its Python API.

现在，让我们在Sublime Text中打开文件，在其中可以看到我们的单曲？ 字符。 因为我们看到了预期的字形，所以我们可以假设Sublime Text使用正确的字符编码将这些字节解码为代码点。 让我们确认通过打开控制台V iew -> Show Conso乐和检查日e vi EW对象崇高文本自曝作为它的Python API的一部分。

>>> view<sublime.View object at 0x1112d7310>

# returns the encoding currently associated with the file>>> view.encoding()'UTF-8'

With a bit of Python knowledge, we can also find the Unicode code point associated with our emoji:

掌握一点Python知识，我们还可以找到与表情符号相关的Unicode代码点：

# Returns the character at the given position>>> view.substr(0)'?'

# ord returns an integer representing the Unicode code point of the character (docs)>>> ord(view.substr(0))128558

# convert code point to hexadecimal, and format with U+>>> print('U+%x' % ord(view.substr(0)))U+1f62e

Again, just as we expected. This illustrates a full traversal of the Unicode rendering chain, which involved:

再次，正如我们所期望的。 这说明了Unicode渲染链的完整遍历，其中涉及：

• reading the file as a sequence of UTF-8 encoded bytes.
  以UTF-8编码字节序列读取文件。
• decoding the bytes into a Unicode code point.
  将字节解码为Unicode代码点。
• rendering the glyph associated with the code point.
  渲染与代码点关联的字形。

So far, so good ?.

到目前为止，一切都很好 ？。

不同的字节，相同的表情符号 (Different Bytes, Same Emoji)

Aside from being my favorite text editor, I chose Sublime Text for this example because it allows for easy experimentation with character encodings.

除了成为我最喜欢的文本编辑器之外，我在此示例中选择了Sublime Text，因为它可以轻松地进行字符编码实验。

We can now save the file using a different character encoding. To do so, click File -> Save with Encoding -> UTF-16 BE. (Very briefly, UTF-16 is an alternative character encoding of the Unicode character set. Instead of encoding the most common characters using one byte, like UTF-8, UTF-16 encodes every point from 1–65536 using two bytes. Code points greater than 65536, like our emoji, are encoded using surrogate pairs. The BE stands for Big Endian).

现在，我们可以使用其他字符编码来保存文件。 为此，请单击File -> Save with Encoding -> UTF -16 BE。 (非常方便， UTF-16是Unicode字符集的替代字符编码。UTF -16不会使用一个字节来编码最常见的字符(如UTF-8)，而是使用两个字节来编码1–65536中的每个点。大于65536的点(如我们的emoji表情)是使用代理对编码的。BE代表Big Endian)。

When we use hexdump to inspect the file again, we see that byte contents have changed.

当我们使用hexdump再次检查文件时，我们看到字节内容已更改。

# (before: UTF-8)j|encoding: hexdump emoji.txt0000000 f0 9f 98 ae 0000004

# (after: UTF-16 BE)j|encoding: hexdump emoji.txt0000000 d8 3d de 2e0000004

Back in Sublime Text, we still see the same ? character staring at us. Saving the file with a different character encoding might have changed the actual contents of the file, but it also updated Sublime Text’s internal representation of how to interpret those bytes. We can confirm by firing up the console again.

回到Sublime Text，我们仍然看到相同的内容吗？ 角色盯着我们。 使用不同的字符编码保存文件可能会更改文件的实际内容，但同时也会更新Sublime Text关于如何解释这些字节的内部表示形式。 我们可以通过再次启动控制台来确认。

>>> view.encoding()'UTF-16 BE'

From here on up, everything else is the same.

从这里开始，其他所有内容都是相同的。

>>> view.substr(0)'?'

>>> ord(view.substr(0))128558

>>> print('U+%x' % ord(view.substr(0)))U+1f62e

The bytes may have changed, but the code point did not — and the emoji remains the same.

字节可能已更改，但是代码点没有更改，并且表情符号保持不变。

相同的字节，但是什么 (Same Bytes, But What The đŸ˜®)

Time for some encoding “fun”. First, let’s re-encode our file using UTF-8, because it makes for a better example.

是时候进行一些编码“乐趣”了。 首先，让我们使用UTF-8重新编码文件，因为它可以提供更好的示例。

Let’s now go ahead use Sublime Text to re-open an existing file using a different character encoding. Under File -> Reopen with Encoding, click Vietnamese (Windows 1258), which turns our emoji character into the following four nonsensical characters: đŸ˜®.

现在，让我们继续使用Sublime Text，使用不同的字符编码重新打开现有文件。 在“ File -> Reopen with Encod ck Vietnamese (Windows 12 58)，这会将我们的表情符号字符转换为以下四个无意义的字符：đŸ〜®。

When we click “Reopen with Encoding”, we aren’t changing the actual byte contents of the file, but rather, the way Sublime Text interprets those bytes. Hexdump confirms the bytes are the same:

当我们单击“使用编码重新打开”时，我们并没有更改文件的实际字节内容，而是更改Sublime Text解释这些字节的方式。 Hexdump确认字节相同：

j|encoding: hexdump emoji.txt0000000 f0 9f 98 ae0000004

To understand why we see these nonsensical characters, we need to consult the Windows-1258 code page, which is a mapping of bytes to a Vietnamese language character set. (Think of a code page as the table produced by a character encoding). As this code page contains a character set with less than 255 characters, each character’s code points can be expressed as a decimal number between 0 and 255, which in turn can all be encoded using 1 byte.

要了解为什么我们会看到这些荒谬的字符，我们需要查阅Windows-1258代码页，该页是字节到越南语字符集的映射。 (将代码页视为由字符编码产生的表)。 由于此代码页包含的字符集少于255个字符，因此每个字符的代码点都可以表示为0到255之间的十进制数字，而这些数字又可以全部使用1个字节进行编码。

Because our single ? emoji requires 4 bytes to encode using UTF-8, we now see 4 characters when we interpret the file with the Windows-1258 encoding.

因为我们单身？ 表情符号需要4个字节才能使用UTF-8进行编码，现在使用Windows-1258编码解释文件时，我们会看到4个字符。

A wrong choice of character encoding has a direct impact on what we can see and comprehend by garbling characters into an incomprehensible mess.

字符编码的错误选择会导致字符混乱，从而对我们所看到和理解的内容产生直接影响。

Now, onto the “fun” part, which I include to add some color to Unicode and why it exists. Before Unicode, there were many different code pages such as Windows-1258 in existence, each with a different way of mapping 1 byte’s worth of data into 255 characters. Unicode was created in order to incorporate all the different characters of the all the different code pages into one system. In other words, Unicode is a superset of Windows-1258, and each character in the Windows-1258 code page has a Unicode counterpart.

现在，进入“有趣”部分，其中包括为Unicode添加颜色以及其存在的原因。 在Unicode之前，存在许多不同的代码页，例如Windows-1258，每个代码页都有不同的方式将1个字节的数据映射为255个字符。 创建Unicode是为了将所有不同代码页的所有不同字符合并到一个系统中 。 换句话说，Unicode是Windows-1258的超集，并且Windows-1258代码页中的每个字符都有一个Unicode对应字符 。

In fact, these Unicode counterparts are what allows Sublime Text to convert between different character encodings with a click of a button. Internally, Sublime Text still represents each of our “Windows-1258 decoded” characters as a Unicode code point, as we see below when we fire up the console:

实际上，这些Unicode对应内容使Sublime Text可以通过单击按钮在不同的字符编码之间进行转换。 在内部，Sublime Text仍将我们的每个“ Windows-1258解码”字符都表示为Unicode代码点，如下图所示，当启动控制台时：

>>> view.encoding()'Vietnamese (Windows 1258)'

# Python 3 strings are "immutable sequences of Unicode code points">>> type(view.substr(0))<class 'str'>

>>> view.substr(0)'đ'>>> view.substr(1)'Ÿ'>>> view.substr(2)'˜'>>> view.substr(3)'®'

>>> ['U+%04x' % ord(view.substr(x)) for x in range(0, 4)]['U+0111', 'U+0178', 'U+02dc', 'U+00ae']

This means that we can re-save our 4 nonsensical characters using UTF-8. I’ll leave this one up to you — if you do so, and can correctly predict the resulting hexdump of the file, then you’ve successfully understood the key concepts behind Unicode, code points, and character encodings. (Use this UTF-8 code page. Answer can be found at the very end of this article. ).

这意味着我们可以使用UTF-8重新保存4个荒谬的字符。 我将把这个问题留给您-如果您这样做，并且可以正确预测文件的结果hexdump ，那么您已经成功理解了Unicode，代码点和字符编码背后的关键概念。 ( 使用此UTF-8代码页 。答案可以在本文的最后找到。)

结语 (Wrapping up)

Working effectively with Unicode involves always knowing what level of the rendering chain you are operating on. It means always asking yourself: what do I have? Under the hood, glyphs are nothing but code points. If you are working with code points, know that those code points must be encoded into bytes with a character encoding. If you have a sequence of bytes representing text, know that those bytes are meaningless without knowing the character encoding that was used create those bytes.

有效地使用Unicode涉及始终知道要在哪个级别的渲染链上进行操作。 这意味着总是问自己：我有什么？ 在幕后，字形不过是代码点。 如果您使用代码点，请知道必须使用字符编码将这些代码点编码为字节。 如果您有表示文本的字节序列，请知道这些字节是没有意义的，而无需知道用于创建这些字节的字符编码。

As with any computer science topic, the best way to learn about Unicode is to experiment. Enter characters, play with character encodings, and make predictions that you verify using hexdump. While I hope this article explains everything you need to know about Unicode, I will be more than happy if it merely sets you up to run your own experiments.

与任何计算机科学主题一样，学习Unicode的最佳方法是进行实验。 输入字符，使用字符编码播放，并进行预测，并使用hexdump验证。 尽管我希望本文能够解释您需要了解的有关Unicode的所有知识，但如果它使您能够运行自己的实验，我会感到非常高兴。

Thanks for reading! ?

谢谢阅读！ ？

回答： (Answer:)

j|encoding: $ hexdump emoji.txt0000000 c4 91 c5 b8 cb 9c c2 ae0000008

翻译自: https://www.freecodecamp.org/news/a-beginner-friendly-guide-to-unicode-d6d45a903515/