Linguistic Sorting and Matching

Collation(排序规则)用于决定字符串（字符值）的顺序，Oracle一般使用order来表示collation

The process of determining the mutual ordering of strings (character values) is called a collation. For any two strings, the collation defines whether the strings are equal or whether one precedes the other in the sorting order. In the Oracle documentation, the term sort is often used in place of collation.

Oracle collation分两种类型:

binary collation: 根据字符的编码排序，它是最快的排序类型，常用于英文字母排序，其它字符也可以用，但可能导致排序无意义
linguistic collation: 依赖于语言的排序，它又分为：

a. Monolingual: 用于某一种语言的排序

b. Multilingual：用于多语言间进行排序

c. Unicode Collation Algorithm (UCA)：用于多语言间进行排序，建议使用

While monolingual collation achieves a linguistically correct order for a single language, multilingual collation and UCA collation are designed to handle many languages at the same time. Furthermore, UCA collation conforms to the Unicode Collation Algorithm (UCA) that is a Unicode standard and is fully compatible with the international collation standard ISO 14651. The UCA standard provides a complete linguistic ordering for all characters in Unicode, hence all the languages around the world. With wide deployment of Unicode application, UCA collation is best suited for sorting multilingual data.

（一）Binary Collation

根据字符的编码排序，它是最快的排序类型，常用于英文字母排序

One way to sort character data is based on the numeric values of the characters defined by the character encoding scheme. This is called a binary collation. Binary collation is the fastest type of sort. It produces reasonable results for the English alphabet because the ASCII and EBCDIC standards define the letters A to Z in ascending numeric value.

ASCII与EBCDIC的大小写排序是相反的

In the ASCII standard, all uppercase letters appear before any lowercase letters. In the EBCDIC standard, the opposite is true:all lowercase letters appear before any uppercase letters.

When characters used in other languages are present, a binary collation usually does not produce reasonable results. For example, an ascending ORDER BY query returns the character strings ABC, ABZ, BCD, ÄBC, when Ä has a higher numeric value than B in the character encoding scheme. A binary collation is not usually linguistically meaningful for Asian languages that use ideographic characters.

（二）Linguistic Collation

Oracle会根据设置(NLS_SORT)将字符串转为对应的collation key进行二进制比较

When the comparison conditions =, !=, >, <, >=, <=, BETWEEN, NOT BETWEEN, IN, NOT IN, the query clauses ORDER BY or GROUP BY, or the aggregate function COUNT(DISTINCT) are evaluated according to linguistic rules specified by NLS_SORT, the compared argument values are first transformed to binary values calledcollation keys and then compared byte by byte, like RAW values.

不同的类型collation使用不同策略的collation key:

If a monolingual collation is applied, collation keys contain concatenated major values for characters of the source value followed by concatenated minor values for those characters.

If a multilingual collation is applied, collation keys contain concatenated primary, then secondary, and then tertiary values.

If a UCA collation is applied, collation keys contain concatenated primary, secondary, tertiary, and possibly quaternary values.

是否区分大小写是通过忽略比较层级实现的

The case-insensitive and accent-insensitive multilingual and UCA collations may omit quaternary, tertiary, and secondary values.

Monolingual Collation

Monolingual collation将比较字符转为major value与minor value，它们存于数据库的major table与minor table中，通过两个值进行比较

Oracle Database compares character strings in two steps for monolingual collation：

The first step compares the major value of the entire string from a table of major values. Usually, letters with the same appearance have the same major value.

The second step compares the minor value from a table of minor values.

The major and minor values are defined by Oracle Database. Oracle Database defines letters with diacritic and case differences as having the same major value but different minor values.

Each major table entry contains the Unicode code point and major value for a character. The Unicode code point is a 16-bit binary value that represents a character.

The following table illustrates sample values for sorting a, A, ä, Ä, and b.

Monolingual collation不能用于数据库字符集为非Unicode的多字节字符集

Monolingual collation isnot available for non-Unicode multibyte database character sets. If a monolingual collation is specified when the database character set is non-Unicode multibyte, then the default sort order is the binary sort order of the database character set. One exception is UNICODE_BINARY. This collation is available for all character sets.

Multilingual Collation

Oracle Database provides multilingual collation so that you cansort data in more than one language in one sort. This is useful for regions or languages that have complex sorting rules and for multilingual databases.

For Asian language data or multilingual data, Oracle Database provides a sorting mechanism based on the ISO 14651 standard. For example, Chinese characters can be ordered by the number of strokes, PinYin, or radicals.

For example, Oracle Database supports a monolingual French sort (FRENCH), but you can specify a multilingual French collation (FRENCH_M). _M represents the ISO 14651 standard for multilingual sorting. The sorting order is based on the GENERIC_M sorting order and can sort diacritical marks from right to left.

Multilingual linguistic sort is usually used if the tables contain multilingual data.

If the tables contain only French, then a monolingual French sort might have better performance because it uses less memory. It uses less memory because fewer characters are defined in a monolingual French sort than in a multilingual French sort. There is a trade-off between the scope and the performance of a sort.

Multilingual Collation Levels

Multilingual Collation不同于Monolingual collation，这是通过以下三个层级进行比较的

Primary Level Collation

primary level用于区别基本字符，根据语言环境决定字符的基本顺序

A primary level collation distinguishes between base letters, such as the difference between characters a and b. It is up to individual locales to define whether a is before b, b is before a, or if they are equal. The binary representation of the characters is completely irrelevant.

If a character is an ignorable character, then it is assigned a primary level order (or weight) of zero, which means it is ignored at the primary level. Characters that are ignorable on other levels are given an order of zero at those levels.

如下面字符在primary level下，bat永远在bet之前，但不同bat是同等级顺序的

For example, at the primary level, all variations of bat come before all variations of bet. The variations of bat can appear in any order, and the variations of bet can appear in any order:

Bat

bat

BAT

BET

Bet

bet

Secondary Level Collation

secondary level用于进行一步区分diacritics(变音符号)

A secondary level collation distinguishes between base letters (the primary level collation) before distinguishingbetween diacritics on a given base letter. For example, the character Ä differs from the character A only because it has a diacritic. Thus, Ä and A are the same on the primary level because they have the same base letter (A) but differ on the secondary level.

The following list has been sorted on the primary level (resume comes before resumes) and on the secondary level (strings without diacritics come before strings with diacritics):

resume

résumé

Résumé

Resumes

resumes

résumés

Tertiary Level Collation

A tertiary level用于进行一步区分大小写以及特殊字符

A tertiary level collation distinguishes between base letters (primary level collation), diacritics (secondary level collation), and case (upper case and lower case). It can also include special characters such as +, -, and *.

The following are examples of tertiary level collations:

Characters a and A are equal on the primary and secondary levels but different on the tertiary level because they have different cases.
Characters ä and A are equal on the primary level and different on the secondary and tertiary levels.
The primary and secondary level orders for the dash character - is 0. That is, it is ignored on the primary and secondary levels. If a dash is compared with another character whose primary level weight is nonzero, for example, u, then no result for the primary level is available because u is not compared with anything. In this case, Oracle Database finds a difference between - and u only at the tertiary level.

The following list has been sorted on the primary level (resume comes before resumes) and on the secondary level (strings without diacritics come before strings with diacritics) and on the tertiary level (lower case comes before upper case):

resume

Resume

résumé

Résumé

resumes

Resumes

résumés

Résumés

3. UCA Collation

可以认为Oracle从Monolingual Collation到Multilingual Collation再到12.2的UCA collation是一个发展过程，它与字符集一样，Oracle建议使用UCA collation.

UCA使用一个表(DUCET)定义了所有语言默认不同的排序顺序

Unicode Collation Algorithm (UCA) is a Unicode standard that is fully compatible with the international collation standard ISO 14651. UCA defines aDefaultUnicode Collation Element Table (DUCET) that provides a reasonable default ordering for all languages that are not tailored. To achieve the correct ordering for a particular language, DUCET can be tailored(定制)to meet the linguistic requirements for that language. There are tailorings of DUCET for various languages provided in the Unicode Common Locale Data Repository.

Oracle Database provides UCA collation that fully conforms to UCA 7.0 as of Oracle Database 12c Release 2 (12.2). In addition to the collation based on DUCET, it provides tailored collations for a number of commonly used languages. For example, you can specify UCA collation UCA0700_SCHINESE to sort multilingual data containing Simplified Chinese. The collation will make Simplified Chinese data appear in the PinYin order.

UCA Comparison Levels

与Multilingual Collation类似UCA也是通过多层级来进行比较的，但它有4个层级：

Similar to multilingual collation, UCA collations employ a multilevel comparison algorithm to evaluate characters. This can go up to four levels of comparison:

Primary Level

The primary level is used to distinguish between base letters, which is similar to the comparison used in the primary level collation of the multilingual collation.

Secondary Level

The secondary level is used to distinguish between diacritics if base letters are the same, which is similar to what is used in the secondary level collation of the multilingual collation to distinguish between diacritics.

Tertiary Level

The tertiary level is used to distinguish between cases on a given base letter with the same diacritic, which is similar to what is used in the tertiary level collation of the multilingual collation to distinguish between cases. Moreover, UCA DUCET collation treats punctuations with primary or quaternary significance based on how variable characters are weighted, which is different from the tertiary level collation of the multilingual collation that treat punctuations with tertiary level of significance.

Quaternary Level

The quaternary level is used to distinguish variable characters from other characters, if variable characters are weighted as shifted. It is also used to distinguish Hiragana from Katakana with the same base and case. An example is illustrated in the following figure.

（三）Case-Insensitive and Accent-Insensitive Linguistic Collation

Oracle提供了两种类型的不敏感case-insensitive(口音不敏感，如ä与a是相同顺序)与 accent-insensitive(大小写不敏感)：

Oracle Database provides case-insensitive and accent-insensitive options for collation. It provides the following types of linguistic collations:

Monolingual collations that use information about base letters, diacritics, and punctuation标点符号, but not case, and multilingual and UCA collations that use information about base letters and diacritics, but not case or punctuation. This type of sort is called case-insensitive.
Monolingual collations that use information aboutbase letters and punctuation only, and multilingual and UCA collations that use information about base letters only. This type of sort is calledaccent-insensitive. (Accent is another word for diacritic.) Like case-insensitive sorts, an accent-insensitive sort does not use information about case.

默认collation为accent-sensitive与case-sensitive, 以_CI结尾的collation name表示accent-insensitive, 以_AI结尾的collation name表示accent-insensitive与case-insensitive. 使用时使用NLS_SORT指定相应collation即可

Use the NLS_SORT session parameter to specify a case-insensitive or accent-insensitive collation:

Append _CI to an Oracle Database collation name for a case-insensitive collation.

Append _AI to an Oracle Database collation name for an accent-insensitive and case-insensitive collation.

Binary collation can also be case-insensitive or accent-insensitive. When you specify BINARY_CI as a value for NLS_SORT, it designates a collation that is accent-sensitive and case-insensitive. BINARY_AI designates an accent-insensitive and case-insensitive binary collation.

Example 5-4 Linguistic Collation Using Base Letters, Diacritics, Punctuation, and Case Information

The following list has been sorted using information about base letters, diacritics, punctuation, and case:

blackbird

black bird

black-bird

Blackbird

Black-bird

blackbîrd

bläckbird

默认是都敏感的(包含diacritics的字符排在了最后)

Example 5-5 Case-Insensitive Linguistic Collation

The following list has been sorted using informationabout base letters, diacritics, and punctuation, ignoring case:

black bird

black-bird

Black-bird

blackbird

Blackbird

blackbîrd

bläckbird

black-bird and Black-bird have the same value in the collation, because the only different between them is case. They could appear interchanged in the list. Blackbird and blackbird also have the same value in the collation and could appear interchanged in the list.

Example 5-6 Accent-Insensitive Linguistic Collation

The following list has been sorted using information about base letters only. No information about diacritics, punctuation, or case has been used.

blackbird

bläckbird

blackbîrd

Blackbird

BlackBird

Black-bird

Black bird

（三）Using Linguistic Comparisons

12.2开始UCA collations有了参数的概念，这些参数并不能指定，而是固定到collation的名字中，不同名字后缀表示不同参数的collation

The following table illustrates the collation parameters and options that are supported in UCA collations starting with Oracle Database 12c Release 2 (12.2).

The parameter strength represents UCA comparison level.

The parameter alternate controls how variable characters are weighted.

The parameter backwards controls if diacritics are to be sorted backward.

The parameter hiraganaQuaternary is applicable to the UCA collations for the Japanese language only. It has no effect on other collations. If it is set to “on” (_HY), then the corresponding Hiragana and Katakana characters have different quaternary weights. Otherwise, they have the same weights. The hiraganaQuaternary parameter is deprecated in UCA 7.0.

You can configure the preceding four UCA parameters using the options listed in Table 5-2. The options for the other parameters listed in Table 5-2 are currently fixed based on tailored languages and are not configurable as of Oracle Database 12c.

12.2以后你可以在创建表或视图时指定列使用的collation，如create table t1(id varchar2(10) collate binary); 在排序时会按列的默认collation来排序比较，这部分单列章节详细说明了

Starting with Oracle Database 12c Release2 (12.2), a collation-sensitive operation determines the collation to use from the collations associated with its arguments.

A collation can be declared for a table column or a view column when the column is created. This associated collation is then passed along the column values to the operations processing the column. An operation applies a set of precedence rules to determine the collation to use based on the collations of its arguments. Similarly, an operation returning a character value derives collation for the return value from the collations of its arguments.

但你也可以按之前版本的方式通过NLS_COMP与NLS_SORT参数来决定实际使用collation

这两个参数均为设置初始session的默认设置的，如果客户端设置了NLS_LANG将会覆盖两个参数设置，另外NLS_SORT是从NLS_LANGUAGE分离出来的，如果NLS_SORT未设置则以NLS_LANGUAGE设置为准

NLS_COMP用于决定是否使用NLS_SORT参数：

NLS_COMP = { BINARY(default) | LINGUISTIC | ANSI }

BINARY: Normally, comparisons in the WHERE clause and in PL/SQL blocks is binary unless you specify the NLSSORT function.(如果没使用NLSSORT函数就按binary排序)
LINGUISTIC: Comparisons for all SQL operations in the WHERE clause and in PL/SQL blocks should use the linguistic sort specified in the NLS_SORT parameter.
ANSI: A setting of ANSI is for backward compatibility; in general, you should set NLS_COMP to LINGUISTIC

NLS_SORT = { BINARY | linguistic_definition }

不同SQL或PL/SQL具体可能不同：

Table 5-3 Linguistic Comparison Behavior with NLS_COMP Settings

SQL or PL/SQL Operation	BINARY	LINGUISTIC	ANSI
Set Operators	-	-	-
UNION, INTERSECT, MINUS	Binary	Honors NLS_SORT	Binary
Scalar Functions	-	-	-
DECODE	Binary	Honors NLS_SORT	Binary
INSTRx	Binary	Honors NLS_SORT	Binary
LEAST, GREATEST	Binary	Honors NLS_SORT	Binary
MAX, MIN	Binary	Honors NLS_SORT	Binary
NLS_INITCAP	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
NLS_LOWER	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
NLS_UPPER	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
NLSSORT	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
NULLIF	Binary	Honors NLS_SORT	Binary
REGEXP_COUNT	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
REGEXP_INSTR	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
REGEXP_REPLACE	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
REGEXP_SUBSTR	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
REPLACE	Binary	Honors NLS_SORT	Binary
RTRIM, TRIM, LTRIM	Binary	Honors NLS_SORT	Binary
TRANSLATE	Binary	Honors NLS_SORT	Binary
Conditions	-	-	-
=, !=, >, <, >=, <=	Binary	Honors NLS_SORT	Honors NLS_SORT
BETWEEN, NOT BETWEEN	Binary	Honors NLS_SORT	Honors NLS_SORT
IN, NOT IN	Binary	Honors NLS_SORT	Honors NLS_SORT
LIKE	Binary	Honors NLS_SORT	Binary
REGEXP_LIKE	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
CASE Expression	-	-	-
CASE	Binary	Honors NLS_SORT	Binary
Analytic Function Clauses	-	-	-
DISTINCT	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
OVER(ORDER BY)	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
OVER(PARTITION BY)	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT
Subquery Clauses	-	-	-
DISTINCT, UNIQUE	Binary	Honors NLS_SORT	Binary
GROUP BY	Binary	Honors NLS_SORT	Binary
ORDER BY	Honors NLS_SORT	Honors NLS_SORT	Honors NLS_SORT

使用示例：

SQL> SELECT ename FROM emp1;

ENAME

----------------------

Mc Calla

MCAfee

McCoye

Mccathye

McCafeé

Example 5-10 Binary Comparison Binary Collation

SQL> ALTER SESSION SET NLS_COMP=BINARY;

SQL> ALTER SESSION SET NLS_SORT=BINARY;

SQL> SELECT ename FROM emp1 WHERE ename LIKE 'McC%e';

ENAME

----------------------

McCoye

Example 5-11 Linguistic Comparison Binary Case-Insensitive Collation

SQL> ALTER SESSION SET NLS_COMP=LINGUISTIC;

SQL> ALTER SESSION SET NLS_SORT=BINARY_CI;

SQL> SELECT ename FROM emp1 WHERE ename LIKE 'McC%e';

ENAME

----------------------

McCoye

Mccathye

Example 5-12 Linguistic Comparison Binary Accent-Insensitive Collation

SQL> ALTER SESSION SET NLS_COMP=LINGUISTIC;

SQL> ALTER SESSION SET NLS_SORT=BINARY_AI;

SQL> SELECT ename FROM emp1 WHERE ename LIKE 'McC%e';

ENAME

----------------------

McCoye

Mccathye

McCafeé

Using Linguistic Indexes

When data in multiple languages is stored in the database, you may want applications to collate the data returned from a SELECT...ORDER BY statement according to different collation sequences depending on the language. You can accomplish this without sacrificing performance by using linguistic indexes. Although a linguistic index for a column slows down inserts and updates, it greatly improves the performance of linguistic collation with the ORDER BY clause and the WHERE clause.

Linguistic indexes本质是函数索引，使用格式：

NLSSORT(column,'NLS_SORT=collation')

The following statement creates an index based on a German collation:

CREATE TABLE my_table(name VARCHAR(20) NOT NULL);

CREATE INDEX nls_index ON my_table (NLSSORT(name, 'NLS_SORT = German'));

The NOT NULL in the CREATE TABLE statement ensures that the index is used.

Supported SQL Operations and Functions for Linguistic Indexes

Linguistic index support is available for the following collation-sensitive SQL operations and SQL functions:

Comparison conditions =, !=, >, <, >=, <=

Range conditions BETWEEN | NOT BETWEEN

IN | NOT IN

ORDER BY

GROUP BY

LIKE (LIKE, LIKE2, LIKE4, LIKEC)

DISTINCT

UNIQUE

UNION

INTERSECT

MINUS

The SQL functions in the following list cannot utilize linguistic index:

INSTR (INSTR, INSTRB, INSTR2, INSTR4, INSTRC)

MAX

MIN

REPLACE

TRIM

LTRIM

RTRIM

TRANSLATE

Linguistic Indexes for Multiple Languages

There are four ways to build linguistic indexes for data in multiple languages:

Build a linguistic index for each language that the application supports. This approach offers simplicity but requires more disk space. For each index, the rows in the language other than the one on which the index is built are collated together at the end of the sequence. The following example builds linguistic indexes for French and German.

CREATE INDEX french_index ON employees (NLSSORT(employee_id, 'NLS_SORT=FRENCH'));

CREATE INDEX german_index ON employees (NLSSORT(employee_id, 'NLS_SORT=GERMAN'));

Oracle Database chooses the index based on the NLS_SORT session parameter or the arguments of the NLSSORT function specified in the ORDER BY clause. For example, if the NLS_SORT session parameter is set to FRENCH, then Oracle Database uses french_index. When it is set to GERMAN, Oracle Database uses german_index.

Build a single linguistic index for all languages. This requires a language column (LANG_COL in "Example: Setting Up a French Linguistic Index") to be used as a parameter of the NLSSORT function. The language column contains NLS_LANGUAGE values for the data in the column on which the index is built. The following example builds a single linguistic index for multiple languages. With this index, the rows with the same values for NLS_LANGUAGE are sorted together.

CREATE INDEX i ON t (LANG_COL, NLSSORT(col, 'NLS_SORT=' || LANG_COL));

Queries choose an index based on the argument of the NLSSORT function specified in the ORDER BY clause.

Build a single linguistic index for all languagesusing one of the multilingual collations such as GENERIC_M or FRENCH_M. These indexes sort characters according to the rules defined in ISO 14651. For example:

CREATE INDEX i ON t (NLSSORT(col, 'NLS_SORT=GENERIC_M'));

Build a single linguistic index for all languages using one of the UCA collations such as UCA0700_ORADUCET or UCA0700_CFRENCH. These indexes sort characters in the order conforming to ISO 14651 and UCA 7.0. For example:

CREATE INDEX i ON t (NLSSORT(col, 'NLS_SORT=UCA0700_ORADUCET'));

Requirements for Using Linguistic Indexes

Set NLS_SORT Appropriately

The NLS_SORT parameter should indicate the linguistic definition you want to use for the linguistic collation. If you want a French linguistic collation order, then NLS_SORT should be set to FRENCH. If you want a German linguistic collation order, then NLS_SORT should be set to GERMAN.

There are several ways to set NLS_SORT. You should set NLS_SORT as a client environment variable so that you can use the same SQL statements for all languages. Different linguistic indexes can be used when NLS_SORT is set in the client environment.

Specify NOT NULL in a WHERE Clause If the Column Was Not Declared NOT NULL

When you want to use the ORDER BY column_name clause with a column that has a linguistic index, include a WHERE clause like the following example:

WHERE NLSSORT(column_name) IS NOT NULL

This WHERE clause is not necessary if the column has already been defined as a NOT NULL column in the schema.

Use a Tablespace with an Adequate Block Size

A collation key created from a character value is usually a few times longer than this value. The actual length expansion depends on the particular collation in use and the content of the source value, with the UCA-based collations expanding the most.

When creating a linguistic index, Oracle Database first calculates the estimated maximum size of the index key by summing up the estimated maximum sizes of the collation keys (NLSSORT results) for each of the character columns forming the index key. In this calculation, the maximum size of a collation key for a character column with the maximum byte length n is estimated to be n*21+5 for UCA-based collations and n*8+10 for other collations.

The large expansion ratios can yield large maximum index key sizes, especially for composite (multicolumn) keys. At the same time, the maximum key size of an index cannot exceed around 70% of the block size of the tablespace containing the index. If it does, an ORA-1450 error is reported. To avoid this error, you should store the linguistic index in a tablespace with an adequate block size, which may be larger than the default block size of your database. A suitable tablespace can be created with the CREATE TABLESPACE statement, provided the initialization parameter DB_nK_CACHE_SIZE corresponding to the required block size n has been set appropriately.

Example: Setting Up a French Linguistic Index

The following example shows how to set up a French linguistic index. You may want to set NLS_SORT as a client environment variable instead of using the ALTER SESSION statement.

ALTER SESSION SET NLS_SORT='FRENCH';

CREATE INDEX test_idx ON test4(NLSSORT(name, 'NLS_SORT=FRENCH'));

SELECT * FROM test4 ORDER BY col;

ALTER SESSION SET NLS_COMP=LINGUISTIC;

SELECT * FROM test4 WHERE name > 'Henri';