Zookeeper 服务器端和客户端扩大节点数据1M大小限制

Zookeeper中每个结点默认的数据量上限是1M，如果需要存入大于1M的数据量，则要修改jute.maxbuffer参数，先来看下zookeeper中这个参数的意义

jute.maxbuffer： 默认值1048575，单位字节，用于配置单个数据节点（ZNode）上可以存储的最大数据大小。需要注意的是，在修改该参数的时候，需要在zookeeper集群的所有服务端以及客户端上设置才能生效。

接下来，我们就来看看，怎么设置这个参数吧。

服务器端：
如果服务器端没有修改该参数，直接写入过大数据会抛错：org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /test

解决办法：

zkServer.sh 新增-Djute.maxbuffer配置，这边以10M为例，具体大小需按实际情况修改（ZOO_USER_CFG为修改过部分的关键词）：

 #!/usr/bin/env bash# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements.  See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License.  You may obtain a copy of the License at##     http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.## If this scripted is run out of /usr/bin or some other system bin directory# it should be linked to and not copied. Things like java jar files are found# relative to the canonical path of this script.## use POSTIX interface, symlink is followed automaticallyZOOBIN="${BASH_SOURCE-$0}"ZOOBIN="$(dirname "${ZOOBIN}")"ZOOBINDIR="$(cd "${ZOOBIN}"; pwd)"if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then. "$ZOOBINDIR/../libexec/zkEnv.sh"else. "$ZOOBINDIR/zkEnv.sh"fi# See the following page for extensive details on setting# up the JVM to accept JMX remote management:# http://java.sun.com/javase/6/docs/technotes/guides/management/agent.html# by default we allow local JMX connectionsZOO_USER_CFG="-Djute.maxbuffer=10240000"if [ "x$JMXLOCALONLY" = "x" ]thenJMXLOCALONLY=falsefiif [ "x$JMXDISABLE" = "x" ] || [ "$JMXDISABLE" = 'false' ]thenecho "ZooKeeper JMX enabled by default" >&2if [ "x$JMXPORT" = "x" ]then# for some reason these two options are necessary on jdk6 on Ubuntu#   accord to the docs they are not necessary, but otw jconsole cannot#   do a local attachZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=$JMXLOCALONLY org.apache.zookeeper.server.quorum.QuorumPeerMain"elseif [ "x$JMXAUTH" = "x" ]thenJMXAUTH=falsefiif [ "x$JMXSSL" = "x" ]thenJMXSSL=falsefiif [ "x$JMXLOG4J" = "x" ]thenJMXLOG4J=truefiecho "ZooKeeper remote JMX Port set to $JMXPORT" >&2echo "ZooKeeper remote JMX authenticate set to $JMXAUTH" >&2echo "ZooKeeper remote JMX ssl set to $JMXSSL" >&2echo "ZooKeeper remote JMX log4j set to $JMXLOG4J" >&2ZOOMAIN="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=$JMXPORT -Dcom.sun.management.jmxremote.authenticate=$JMXAUTH -Dcom.sun.management.jmxremote.ssl=$JMXSSL -Dzookeeper.jmx.log4j.disable=$JMXLOG4J org.apache.zookeeper.server.quorum.QuorumPeerMain"fielseecho "JMX disabled by user request" >&2ZOOMAIN="org.apache.zookeeper.server.quorum.QuorumPeerMain"fiif [ "x$SERVER_JVMFLAGS"  != "x" ]thenJVMFLAGS="$SERVER_JVMFLAGS $JVMFLAGS"fiif [ "x$2" != "x" ]thenZOOCFG="$ZOOCFGDIR/$2"fi# if we give a more complicated path to the config, don't screw around in $ZOOCFGDIRif [ "x$(dirname "$ZOOCFG")" != "x$ZOOCFGDIR" ]thenZOOCFG="$2"fiif $cygwinthenZOOCFG=`cygpath -wp "$ZOOCFG"`# cygwin has a "kill" in the shell itself, gets confusedKILL=/bin/killelseKILL=killfiecho "Using config: $ZOOCFG" >&2case "$OSTYPE" in*solaris*)GREP=/usr/xpg4/bin/grep;;*)GREP=grep;;esacif [ -z "$ZOOPIDFILE" ]; thenZOO_DATADIR="$($GREP "^[[:space:]]*dataDir" "$ZOOCFG" | sed -e 's/.*=//')"if [ ! -d "$ZOO_DATADIR" ]; thenmkdir -p "$ZOO_DATADIR"fiZOOPIDFILE="$ZOO_DATADIR/zookeeper_server.pid"else# ensure it exists, otw stop will failmkdir -p "$(dirname "$ZOOPIDFILE")"fiif [ ! -w "$ZOO_LOG_DIR" ] ; thenmkdir -p "$ZOO_LOG_DIR"fi_ZOO_DAEMON_OUT="$ZOO_LOG_DIR/zookeeper.out"case $1 instart)echo  -n "Starting zookeeper ... "if [ -f "$ZOOPIDFILE" ]; thenif kill -0 `cat "$ZOOPIDFILE"` > /dev/null 2>&1; thenecho $command already running as process `cat "$ZOOPIDFILE"`. exit 0fifinohup "$JAVA" "$ZOO_USER_CFG" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &if [ $? -eq 0 ]thencase "$OSTYPE" in*solaris*)/bin/echo "${!}\\c" > "$ZOOPIDFILE";;*)/bin/echo -n $! > "$ZOOPIDFILE";;esacif [ $? -eq 0 ];thensleep 1echo STARTEDelseecho FAILED TO WRITE PIDexit 1fielseecho SERVER DID NOT STARTexit 1fi;;start-foreground)ZOO_CMD=(exec "$JAVA")if [ "${ZOO_NOEXEC}" != "" ]; thenZOO_CMD=("$JAVA")fi"${ZOO_CMD[@]}" "$ZOO_USER_CFG" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG";;print-cmd)echo "\"$JAVA\" -Dzookeeper.log.dir=\"${ZOO_LOG_DIR}\" -Dzookeeper.root.logger=\"${ZOO_LOG4J_PROP}\" -cp \"$CLASSPATH\" $JVMFLAGS $ZOOMAIN \"$ZOOCFG\" > \"$_ZOO_DAEMON_OUT\" 2>&1 < /dev/null";;stop)echo -n "Stopping zookeeper ... "if [ ! -f "$ZOOPIDFILE" ]thenecho "no zookeeper to stop (could not find file $ZOOPIDFILE)"else$KILL -9 $(cat "$ZOOPIDFILE")rm "$ZOOPIDFILE"echo STOPPEDfiexit 0;;upgrade)shiftecho "upgrading the servers to 3.*""$JAVA" "$ZOO_USER_CFG" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $JVMFLAGS org.apache.zookeeper.server.upgrade.UpgradeMain ${@}echo "Upgrading ... ";;restart)shift"$0" stop ${@}sleep 3"$0" start ${@};;status)# -q is necessary on some versions of linux where nc returns too quickly, and no stat result is outputclientPortAddress=`$GREP "^[[:space:]]*clientPortAddress[^[:alpha:]]" "$ZOOCFG" | sed -e 's/.*=//'`if ! [ $clientPortAddress ]thenclientPortAddress="localhost"ficlientPort=`$GREP "^[[:space:]]*clientPort[^[:alpha:]]" "$ZOOCFG" | sed -e 's/.*=//'`STAT=`"$JAVA" "$ZOO_USER_CFG" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $JVMFLAGS org.apache.zookeeper.client.FourLetterWordMain \$clientPortAddress $clientPort srvr 2> /dev/null    \| $GREP Mode`if [ "x$STAT" = "x" ]thenecho "Error contacting service. It is probably not running."exit 1elseecho $STATexit 0fi;;*)echo "Usage: $0 {start|start-foreground|stop|restart|status|upgrade|print-cmd}" >&2esac

客户端：
改完zkServer.sh重启zookeeper后超过1M数据可以写进去了，但是从客户端读的时候又抛错了java.io.IOException: Unreasonable length = 1560001

zkCli.sh

接下来就要去改客户端的配置，如果是直接使用zookeeper自带的zkCli.sh读取数据，那么要在zkCli.sh中新增-Djute.maxbuffer配置，这边以10M为例，具体大小需按实际情况修改，（ZOO_USER_CFG为修改过部分的关键词）：

 #!/usr/bin/env bash# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements.  See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License.  You may obtain a copy of the License at##     http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.## This script cleans up old transaction logs and snapshots### If this scripted is run out of /usr/bin or some other system bin directory# it should be linked to and not copied. Things like java jar files are found# relative to the canonical path of this script.## use POSTIX interface, symlink is followed automaticallyZOOBIN="${BASH_SOURCE-$0}"ZOOBIN="$(dirname "${ZOOBIN}")"ZOOBINDIR="$(cd "${ZOOBIN}"; pwd)"ZOO_USER_CFG="-Djute.maxbuffer=10240000"if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then. "$ZOOBINDIR"/../libexec/zkEnv.shelse. "$ZOOBINDIR"/zkEnv.shfi"$JAVA" "$ZOO_USER_CFG" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" \-cp "$CLASSPATH" $CLIENT_JVMFLAGS $JVMFLAGS \org.apache.zookeeper.ZooKeeperMain "$@"

代码块中

如果是在代码中读取，那么要把修改后的jute.maxbuffer属性值重新写入，我这边是在java代码中使用zookeeper-3.4.10.jar包，先来看下这个包中是怎么初始化jute.maxbuffer值的

maxbuffer会先从系统属性中读取jute.maxbuffer值（System.getProperty()）,如果取不到则赋值1048575
知道maxbuffer的取值逻辑后，我们就可以对症下药啦，只要在初始化zookeeper客户端之前，往系统属性中加入修改后的jute.maxbuffer键值对就可以啦

最后再啰嗦一点，zookeeper上不宜存储太多数据，所以jute.maxbuffer的值需要按具体情况定哦

参考：
《从Paxos到ZooKeeper》
https://blog.csdn.net/hopingwhite/article/details/8255979

Zookeeper 服务器端和客户端扩大节点数据1M大小限制相关推荐

C# 视频监控系列（5）：客户端——给服务器端发送字符串和录像(数据捕获)
前言这几天加紧赶工写服务器端的程序,所有系列文章更新较慢,见谅: ) 注意本系列文章限于学习交流,注重过程,由于涉及公司,所以不提供源代码下载,非常抱歉!!但是请大家放心,核心.实现以及其他能够贴 ...
android 如何从服务器端的数据库中拿数据，在客户端显示类？
============问题描述============ android 如何从服务器端的数据库中拿数据,在客户端显示类? ============解决方案1============ 写一个网络访问的 ...
java基于http协议客户端与服务器端的交互,通俗易懂客户端与服务器端交互原理（HTTP数据请求与HTTP响应，包括Servlet部分...
经常看到HTTP客户端与服务器端交互原理的各种版本的文章,但是专业术语太多,且流程过于复杂,不容易消化.于是就按照在 Servlet 里面的内容大致做了一些穿插.本来连 Tomcat 容器和 S ...
【Zookeeper系列】ZooKeeper管理分布式环境中的数据（转）
原文地址:https://www.cnblogs.com/sunddenly/p/4092654.html 引言本节本来是要介绍ZooKeeper的实现原理,但是ZooKeeper的原理比较复杂,它 ...
分布式服务框架 Zookeeper — 管理分布式环境中的数据
FROM: http://www.superwu.cn/2014/11/26/1461 本节本来是要介绍ZooKeeper的实现原理,但是ZooKeeper的原理比较复杂,它涉及到了paxos算法.Z ...
ZooKeeper管理分布式环境中的数据
Reference: http://www.cnblogs.com/wuxl360/p/5817549.html 本节本来是要介绍ZooKeeper的实现原理,但是ZooKeeper的原理比较复杂,它 ...
ZooKeeper学习第五期--ZooKeeper管理分布式环境中的数据
2019独角兽企业重金招聘Python工程师标准>>> 引言本节本来是要介绍ZooKeeper的实现原理,但是ZooKeeper的原理比较复杂,它涉及到了paxos算法.Zab协议 ...
zookeeper的C客户端API介绍及编译测试程序（未完待续11/01，缺测试程序）
文章目录一.初始化.销毁 Zookeeper 句柄 1)初始化zookeeper句柄 2)销毁zookeeper句柄二.辅助函数 1)设置日志等级 2)获取客户端的 session id,只有在客 ...
分布式服务框架 Zookeeper -- 管理分布式环境中的数据
2019独角兽企业重金招聘Python工程师标准>>> 转自:http://www.ibm.com/developerworks/cn/opensource/os-cn-zookee ...

Zookeeper 服务器端和客户端扩大节点数据1M大小限制

Zookeeper 服务器端和客户端扩大节点数据1M大小限制相关推荐

最新文章

热门文章