可伸缩性解密：集群、节点和分片

更新连载中…请关注

Scalability and resilience: clusters,nodes, and shard

Elasticsearch支持根据需要进行扩缩容.这得益于Elasticsearch是原生支持分布式的.可以通过往机器中添加服务器（节点）的方式扩大集群容量从而存储更多数据.Elasticsearch会自动的均一些数据和计算任务给新加入的数据.甚至不需要应用程序参与，Elasticsearch完全知道该怎么把数据均衡到多个节点并且提供良好的可伸缩性和高可用性.集群的节点越多这种操作越顺滑越无感. 就是这么丝滑，堪比丝袜!

Elasticsearch is built to be always available and to scale with your needs. It does this by being distributed by nature. You can add servers (nodes) to a cluster to increase capacity and Elasticsearch automatically distributes your data and query load across all of the available nodes. No need to overhaul your application, Elasticsearch knows how to balance multi-node clusters to provide scale and high availability. The more nodes, the merrier.

为什么能这么丝滑呢?用的什么配方?爆开揉碎一探究竟，在Elasticsearch的内部索引其实是包含一个或多个物理存储分片的逻辑组。每个分片都有它自己的索引数据.Elasticsearch就是通过把索引中的文档划分到多个分片（shard）中，再把划分的分片分到多个节点来实现索引的分布式存储的.当机器发生扩容或缩容时，Elasticsearch自动迁移重新均衡分片，这样就能保证在磁盘损坏或通过新加服务器（节点）扩容时依然可以正常提供服务.

How does this work? Under the covers, an Elasticsearch index is really just a logical grouping of one or more physical shards, where each shard is actually a self-contained index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple nodes, Elasticsearch can ensure redundancy, which both protects against hardware failures and increases query capacity as nodes are added to a cluster. As the cluster grows (or shrinks), Elasticsearch automatically migrates shards to rebalance the cluster.

等等！好像哪里不对,扩容好说，磁盘损坏了上面的分片不都坏了,还能提供正常服务?按读书少，你可不要骗俺啊?

这就要说下分片的类型了，其实有俩种类型的分片: 主分片和副分片(备用分片).在索引中的每个文档隶属于一个主分片.副分片就是主分片的备份.

副分片:

其实我就是传说中的备胎，咿呀咿呀哟!

主分片和副分片通常是不在一个磁盘上的，当发生磁盘损坏时，磁盘上的主分片对应的副分片也就转正了，这样就解释了为什么当磁盘损坏时Elasticsearch仍能提供服务了.另外备胎，不！是副分片也能提供读服务，这样就提高了集群的读取文档的吞吐量.(因为同时可以有多台机器提供文档读取服务)

There are two types of shards: primaries and replicas. Each document in an index belongs to one primary shard. A replica shard is a copy of a primary shard. Replicas provide redundant copies of your data to protect against hardware failure and increase capacity to serve read requests like searching or retrieving a document.

索引的主分片数量需要在创建索引的时候指定，创建后就不能修改了。但副分片的数量在索引创建后还是可以修改地.而且修改副分片数量不会影响正在执行的索引和查询操作. 这就是备胎的份量啊!

The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time, without interrupting indexing or query operations.

分片如此重要，怎么设置分片大小和数量呢?

It depands:

设置索引的分片大小和主分片数量时，有些我们需要权衡取舍地方:

分片越多维护这些分片的开销就越大，分片越大再进行数据均衡时需要移动的数据也越多耗费的时间也越多.

鱼和熊掌不可兼得

There are a number of performance considerations and trade offs with respect to shard size and the number of primary shards configured for an index. The more shards, the more overhead there is simply in maintaining those indices. The larger the shard size, the longer it takes to move shards around when Elasticsearch needs to rebalance a cluster.

查询的分片越小查询每个分片用的时间越少，但也意味需要更多的查询，查询越多消耗也越大，所以有时候分区大点分区数量少点查询可能反而更快.总之一句话，你自己看着办吧.

Querying lots of small shards makes the processing per shard faster, but more queries means more overhead, so querying a smaller number of larger shards might be faster. In short…it depends.

这里给出点建议，仅供参考:

控制分片大小在GB到数十GB.对于时序数据通常可以控制20GB到40GB.
Aim to keep the average shard size between a few GB and a few tens of GB. For use cases with time-based data, it is common to see shards in the 20GB to 40GB range.
避免分片过多，一个节点可以容纳的分片数与可用堆空间成正比.一般来说，每GB堆空间的分片数不应大于20.
Avoid the gazillion shards problem. The number of shards a node can hold is proportional to the available heap space. As a general rule, the number of shards per GB of heap space should be less than 20.

最好的确定分区大小和数量的方式就是在你的应用场景下使用数据和查询测试下.

The best way to determine the optimal configuration for your use case is through testing with your own data and queries.

容灾

In case of disaster

为了保证良好的性能，集群中的节点需要放在同一个网络环境中,因为跨数据中心移动迁移分片会耗费比较长的时间.但是为了高可用，我们又不能把所有鸡蛋都放在一个篮子里.怎么做到当一个数据中心发生重大故障另一个数据中心能够及时的接管，甚至让用户都感觉不到发生了故障呢?可以使用跨集群副本功能也就是CCR.

For performance reasons, the nodes within a cluster need to be on the same network. Balancing shards in a cluster across nodes in different data centers simply takes too long. But high-availability architectures demand that you avoid putting all of your eggs in one basket. In the event of a major outage in one location, servers in another location need to be able to take over. Seamlessly. The answer? Cross-cluster replication (CCR).

跨集群副本可以自动热备份主集群的索引数据到第二集群.如果主集群挂了，第二集群继续对外提供服务.第二集群也可以提供只读的服务这样就可以优选与用户距离近的集群提供服务了.

CCR provides a way to automatically synchronize indices from your primary cluster to a secondary remote cluster that can serve as a hot backup. If the primary cluster fails, the secondary cluster can take over. You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users.

跨集群副本是主从机制.在主集群上的是主，处理所有的写请求。在第二集群上的是从，只提供读服务(只处理读请求).

Cross-cluster replication is active-passive. The index on the primary cluster is the active leader index and handles all write requests. Indices replicated to secondary clusters are read-only followers.

运维

Care And feeding

像其它企业应用系统一样，我们也需要对Elasticsearch进行安全管理和监控.可以通过Kibana提供的控制中心管理、监控Elasticsearch.比如数据回滚、管理索引生命周期啊什么的.

As with any enterprise system, you need tools to secure, manage, and monitor your Elasticsearch clusters. Security, monitoring, and administrative features that are integrated into Elasticsearch enable you to use Kibana as a control center for managing a cluster. Features like data rollups and index lifecycle management help you intelligently manage your data over time

[Elasticsearch]4.可伸缩性解密：集群、节点和分片相关推荐

ElasticSearch优化系列一：集群节点规划
节点职责单一,各司其职 elasticSearch的配置文件中有2个参数:node.master和node.data.这两个参数搭配使用时,能够帮助提供服务器性能. 数据节点node.master: ...
集群节点Elasticsearch升级
集群节点Elasticsearch升级操作流程 1.首先执行Elasticsearch-1.2.2集群的索引数据备份 2.关闭elasticsearch-1.2.2集群的recovery.compr ...
【Elasticsearch】ELASTICSEARCH集群节点的扩容（移除与添加）
1.概述转载:ELASTICSEARCH集群节点的扩容(移除与添加) 0x01 前言我的elasticsearch集群在刚建立之初只是想用于测试,所以每个节点只有300G的磁盘空间.但后来用在我自 ...
ElasticSearch 深入理解三：集群部署设计
ElasticSearch 深入理解三:集群部署设计 ElasticSearch从名字中也可以知道,它的Elastic跟Search是同等重要的,甚至以Elastic为主要导向. Elastic即可 ...
使用 Fluentd 和 ElasticSearch Stack 实现 Kubernetes 的集群 Logging
经过一段时间的探索,我们先后完成了Kubernetes集群搭建,DNS.Dashboard.Heapster等插件安装,集群安全配置,搭建作为Persistent Volume的CephRBD,以及服 ...
elasticsearch系列八：ES 集群管理（集群规划、集群搭建、集群管理）
一.集群规划搭建一个集群我们需要考虑如下几个问题: 1. 我们需要多大规模的集群? 2. 集群中的节点角色如何分配? 3. 如何避免脑裂问题? 4. 索引应该设置多少个分片? 5. 分片应该设置几个 ...
【Es】Elasticsearch 7.x 新的集群协调层
1.概述转载:https://www.easyice.cn/archives/332 ES 7.x 重构了一个新的集群协调层,他实际上是 Raft 的实现,但并非严格按照 Raft 论文实现,而是做 ...
集群节点数和分片数关系_Elasticsearch最佳实践之分片使用优化
一.遇到的问题与大多数分布式系统一样,Elasticsearch按照一定的Hash规则把用户数据切分成多个分片,然后打散到不同机器进行存储,从而实现大规模数据的分布式存储. cluster.png ...
ElasticSearch的基本用法与集群搭建
一.简介 ElasticSearch和Solr都是基于Lucene的搜索引擎,不过ElasticSearch天生支持分布式,而Solr是4.0版本后的SolrCloud才是分布式版本,Solr的分布式 ...
数据库-Elasticsearch进阶学习笔记（集群、故障、扩容、简繁体、拼音等）
目录集群集群配置单节点集群分布式集群故障转移水平扩容路由计算&分片控制数据CRUD流程写流程读流程更新流程删除流程分词器 IK分词器 Pinyin分词器简繁体转换器 ...

[Elasticsearch]4.可伸缩性解密：集群、节点和分片

可伸缩性解密：集群、节点和分片

分片如此重要，怎么设置分片大小和数量呢?

容灾

运维

[Elasticsearch]4.可伸缩性解密：集群、节点和分片相关推荐

最新文章

热门文章