Elasticsearch重建索引

本文基于Elasticsearch7.x

在聊重建索引之前, 我们先了解下Elasticsearch基本概念与核心原理

我们知道Elasticsearch的索引一旦创建是不可变更的, 如果我们要修改索引的Setting, Mapping, 这时就需要重建索引. Elasticsearch内置了两种重建索引的API:

Update By Query
在现有索引上重建索引.
Reindex
在其他索引上重建索引.

Update By Query

Update By Query API是在现有索引上重建索引, 我们来用实例了解下.

(1) 新增一个blogs文档

PUT /blogs/_doc/1
{"content": "Hadoop is cool","title": "hadoop"
}

(2) 查看Mapping

GET /blogs/_mapping

结果:

{"blogs" : {"mappings" : {"properties" : {"content" : {"type" : "text","fields" : {"keyword" : {"type" : "keyword","ignore_above" : 256}}},"title" : {"type" : "text","fields" : {"keyword" : {"type" : "keyword","ignore_above" : 256}}}}}}
}

(3) 修改Mapping

PUT /blogs/_mapping
{"properties": {"content": {"type": "text","fields": {"english": {"type": "text","analyzer": "english"}}}}
}

为content字段再添加一个子字段english, 其分词器为英语分词器.

(4) 查看Mapping

GET /blogs/_mapping

结果:

{"blogs" : {"mappings" : {"properties" : {"content" : {"type" : "text","fields" : {"english" : {"type" : "text","analyzer" : "english"},"keyword" : {"type" : "keyword","ignore_above" : 256}}},"title" : {"type" : "text","fields" : {"keyword" : {"type" : "keyword","ignore_above" : 256}}}}}}
}

可以看到Mapping结构发生了变化, 但只对后面新增的文档生效, 旧文档的Mapping结构不会改变.

(5) 搜索之前写入的文档

GET /blogs/_search
{"query": {"match": {"content.english": "Hadoop"}}
}

结果:

"hits" : {"total" : {"value" : 0,"relation" : "eq"},"max_score" : null,"hits" : [ ]
}

最终结果证实了旧文档的索引没有发生变化.

(6) Update By Query 重建索引

POST /blogs/_update_by_query?conflicts=proceed

结果:

{"took" : 17,"timed_out" : false,"total" : 1,"updated" : 1,"deleted" : 0,"batches" : 1,"version_conflicts" : 0,"noops" : 0,"retries" : {"bulk" : 0,"search" : 0},"throttled_millis" : 0,"requests_per_second" : -1.0,"throttled_until_millis" : 0,"failures" : [ ]
}

conflicts=proceed表示在重建索引的过程中出现某个文档报错, 不要终止, 继续重建索引.

(7) 再次搜索之前写入的文档

GET /blogs/_search
{"query": {"match": {"content.english": "Hadoop"}}
}

结果:

"hits" : [{"_index" : "blogs","_type" : "_doc","_id" : "1","_score" : 0.2876821,"_source" : {"content" : "Hadoop is cool","title" : "hadoop"}
}

重建索引后旧文档的Mapping才修改.

Reindex

Reindex API是在其他索引上重建索引, 即将一个索引的数据复制到另一个索引中. Reindex APi 要求Mapping中的_source属性设置为enable(默认就是enable), 且Reindex只会复制数据, 不会复制Mapping, Setting结构.

Reindex常用属性

(1) version_type

乐观版本控制机制, 值有四种, 分别是:

internal
默认. 将数据复制到另一个索引的过程中, 如果遇到相同的文档id, 直接覆盖.
external
将数据复制到另一个索引的过程中, 如果遇到相同的文档id, 只有版本号更大才会覆盖.
external_gt
同external.
external_gte
版本号大于等于才会覆盖.

(2) op_type

index
默认, 如果遇到相同文档id的数据, 则覆盖.
create
遇到相同文档id的数据则报错.

(3) conflicts

abort
重建索引过程中出现报错时终止重建索引.
proceed
重建索引过程中出现报错时跳过, 继续重建索引.

实例

我们将Update By Query实例数据copy到一个新的索引中.

(1) 定义Mapping

PUT /blogs_fix
{"mappings": {"properties": {"content": {"type": "text","fields": {"english": {"type": "text","analyzer": "english"}}},"title": {"type": "keyword"}}}
}

(2) reindex

POST _reindex
{"source": {"index": "blogs"},"dest": {"index": "blogs_fix","version_type": "internal","op_type": "index"},"conflicts": "proceed"
}

(3) 查看数据

GET /blogs_fix/_doc/1

结果:

{"_index" : "blogs_fix","_type" : "_doc","_id" : "1","_version" : 1,"_seq_no" : 0,"_primary_term" : 1,"found" : true,"_source" : {"content" : "Hadoop is cool","title" : "hadoop"}
}

使用_alias实现不停机重建索引

在实际场景中, 我们ES中的数据都是来自于MySQL, 当我们修改了某个索引的Mapping或Setting需要重建索引时, 我们可以使用_alias来实现不停机重建索引, 即在重建索引的过程中, 旧索引仍然对外提供服务.

(1) 定义Mapping

PUT /blogs
{"mappings": {"properties": {"content": {"type": "text"},"title": {"type": "text"}}}
}

(2) 设置别名

PUT /blogs/_alias/blogs_v1

(3) 通过别名操作索引

PUT /blogs_v1/_doc/1
{"content": "Hadoop is cool","title": "hadoop"
}GET /blogs_v1/_search

(4) 新建一个索引

我们这个时候发现原来的索引Mapping结构不太合适, 想要进行修改.

PUT /blogs_fix
{"mappings": {"properties": {"content": {"type": "text"},"title": {"type": "keyword"}}}
}

(5) 重建索引

将MySQL中对应的数据索引到Elasticsearch中, 这里使用reindex模拟这个操作.

POST _reindex
{"source": {"index": "blogs"},"dest": {"index": "blogs_fix"},"conflicts": "proceed"
}

(6) 重建索引之后再切换别名

POST /_aliases
{"actions": [{"remove": {"index": "blogs","alias": "blogs_v1"}},{"add": {"index": "blogs_fix","alias": "blogs_v1"}}]
}

(7) 通过别名操作索引

通过使用别名操作索引, 使得重建索引对外无感知, 不影响对外提供服务.

GET /blogs_v1/_search