mongodb log ,warning: chunk is larger than 65203623200 bytes because of key

公司的一台放多媒体文件的MONGODB要转成SHARD，在测试环境里面对gridfs里面的fs.chunks做sharding的时候。

在一段时间后报错如下 :

warning: chunk is larger than 65203623200 bytes because of key { files_id: ObjectId('4e2ea40efa30e751113fc633') }

Tue Sep 27 12:27:11 [conn7] about to log metadata event: { _id: "db-xxx-xxx-xxx-xxx.sky-mobi.com.sh.nj-2011-09-27T04:27:11-4388", server: "db-xxx-xxx-xxx-xxx.sky-mobi.com.sh.nj", clientAddr: "xxx.xxx.xxx.xxx:10751", time: new Date(1317097631976), what: "moveChunk.from", ns: "digoal.fs.chunks", details: { min: { files_id: ObjectId('4e2ea40efa30e751113fc633') }, max: { files_id: ObjectId('4e2ea41afa30e751bd40c633') }, step1: 0, step2: 111, note: "aborted" } }

Tue Sep 27 12:28:34 [conn12] warning: can't move chunk of size (approximately) 97528508 because maximum size allowed to move is 67108864 ns: digoal.fs.chunks { files_id: ObjectId('4e2ea40efa30e751113fc633') } -> { files_id: ObjectId('4e2ea41afa30e751bd40c633') }

Tue Sep 27 12:28:45 [conn7] command admin.$cmd command: { moveChunk: "digoal.fs.chunks", from: "digoal001/xxx.xxx.xxx:xxxx,xxx.xxx.xxx:xxxx,xxx.xxx.xxx:xxxx", to: "digoal004/xxx.xxx.xxx:xxxx,xxx.xxx.xxx:xxxx,xxx.xxx.xxx:xxxx", min: { files_id: ObjectId('4e2ea40efa30e751113fc633') }, max: { files_id: ObjectId('4e2ea41afa30e751bd40c633') }, maxChunkSizeBytes: 67108864, shardId: "digoal.fs.chunks-files_id_ObjectId('4e2ea40efa30e751113fc633')", configdb: "xxx.xxx.xxx:xxxx,xxx.xxx.xxx:xxxx,xxx.xxx.xxx:xxxx" } ntoreturn:1 reslen:109 176ms

从日志中可以看到有一个chunk的SIZE是97528508字节,无法移动,because maximum size allowed to move is 67108864

查询了fs.chunks之后，发现有超过150MB的文件。不过这个警告信息是对的，这么大的CHUNK确实不应该被MOVE。应该先SPLIT到64M以下再被MOVE。否则BALANCE进程会带来严重的性能问题。

如果要修改64M的限制，可以进入mongos/config库

db.settings.find()
{ "_id" : "chunksize", "value" : 64 }

db.settings.update( {"_id" : "chunksize"}, { $set: {"value" : 200 } } )
这将改为200MB.
这个值修改后可能要重启mongos才能生效.

但是推荐还是不要修改。

处理这类事件的手段举例 :

修改chunksize :

> use config

> db.settings.update( {"_id" : "chunksize"}, { $set: {"value" : new_chunk_size_in_mb } } )

Note though that for an existing cluster, it may take some time for the collections to split to that size, if smaller than before, and currently

autosplitting is only triggered if the collection gets new documents or updates.

手工拆分chunk :

The following command splits the chunk where the { _id : 99 }} resides (or would reside if present) in two. The key used as the split point is computed internally and is approximately the key which would divide the chunk in two equally sized new chunks.

> use admin

switched to db admin

> db.runCommand( { split : "test.foo" , find : { _id : 99 } } )

...

The Balancer treats all chunks the same way, regardless if they were generated by a manual or an automatic split.

预split :

In the example below the command splits the chunk where the _id 99 would reside using that key as the split point. Again note that a key need not exist for a chunk to use it in its range. The chunk may even be empty.

> use admin

switched to db admin

> db.runCommand( { split : "test.foo" , middle : { _id : 99 } } )

...

split后的chunk值范围:

["$MinKey", "99")

["99", "$MaxKey")

预move chunk :

> db.printShardingStatus("verbose") 找到属于哪个shard,需要移动到哪个shard

> db.runCommand({ moveChunk : "foo.bar" , find : { hash : "8000" }, to : "shard0000" })

预split和预move chunk的好处 :

1. Chunks will not split until the data reaches a certain minimum amount in size (hundreds of megabytes). Until this occurs balancing and migration will not take place. When the data volume is this small, distributing data between multiple servers is not required anyway. When pre-splitting manually, many chunks can exist even if very little data is present for each chunk initially.

2. "db.runCommand( { split : "test.foo" , middle : { _id : 99 } } ) " , This version of the command allows one to do a data presplitting that is especially useful in a load. If the range and distribution of keys to be data presplitting inserted are known in advance, the collection can be split proportionately to the number of servers using the command above, and the (empty) chunks could be migrated upfront using the moveChunk command.

其他参考资料:

http://groups.google.com/group/mongodb-user/browse_thread/thread/aa73647be2ad9cbe

mongodb log ,warning: chunk is larger than 65203623200 bytes because of key相关推荐

linux c 内核 warning: the frame size of 1040 bytes is larger than 1024 bytes
问题描述: 在编译内核模块驱动时,如果出现如下警告信息: warning: the frame size of 1040 bytes is larger than 1024 bytes.主要是因为内核 ...
warning: the frame size of 1040 bytes is larger than 1024 bytes
问题描述: 在编译内核模块驱动时,如果出现如下警告信息: warning: the frame size of 1040 bytes is larger than 1024 bytes.主要是因为内核 ...
POI在sax模式下读取xlsx报错 Characters larger than 4 bytes are not supported解决办法
使用的POI版本为3.14 在sax模式下读取一个20万行的xlsx报错:org.apache.xmlbeans.XmlException: java.io.CharConversionExcepti ...
Warning: Each record in table should have a unique `key` prop,or set `rowKey` to 解决方法
在vue项目中,使用vue.ant.design中的Table组件会出现一下报错: 列表中的每个记录应该有唯一的"key"支持,或者将"rowKey"设置为唯一 ...
springboot post 请求参数过长请求接口报错 An HTTP line is larger than 4096 bytes
springboot项目post请求参数过长请求接口报错如下: 解决办法将post请求参数值调大: server:tomcat:max-http-form-post-size: 5MB
MongoDB 日志切换（Rotate Log Files）实战
MongoDB 日志切换(Rotate Log Files)实战 1. 在mongo shell下,执行logRotate命令: use admin db.runCommand({logRotate: ...
MongoDB分片实战(三)：性能和优化
插入.查询和更新插入:MongoDB会根据片键和conifg中的信息写入到指定的分片上. 读取:以下内容摘抄自<深入学习MongoDB> 关于读取:上一节故障恢复中已经有所验证. 更新: ...
MongoDB分片实战
1. 集群搭建 1.1环境准备 Linux环境主机 OS 备注 192.168.32.13 CentOS6.3 64位普通PC 192.168.71.43 CentOS6.2 64位服务器,NU ...
MongoDB 文档数据库
文章目录一.MongoDB 基本介绍 1.简介 2.应用场景 3.MongoDB的特点 4.MongoDB的优势 5.体系架构二.MongoDB 索引 1.索引操作 2.索引结构 3.索引特性 4 ...

mongodb log ,warning: chunk is larger than 65203623200 bytes because of key

mongodb log ,warning: chunk is larger than 65203623200 bytes because of key相关推荐

最新文章

热门文章