效果图

环境准备

安装 influxdb

安装我们刚刚下载 influxdb rpm文件

查看默认配置

修改参数

启动 influxdb

查看启动状态

设置基本配置

influxdb 其他命令扩展

安装 jmxtrans

可能遇到的异常

验证jmxtrans是否成功运行

安装 Grafana

安装

influxDB 与 Grafana 监控模板

效果图

先看下

环境准备

JDK：1.8

jmxtrans 安装包

# 我们系统是CentOS，这里选择rpm
https://github.com/downloads/jmxtrans/jmxtrans/jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm
# 其他系统在下面链接中可以找到
https://github.com/jmxtrans/jmxtrans/downloads# 下载源码，后面编译使用
https://github.com/jmxtrans/jmxtrans/releases
# 我这里使用的271版本
https://github.com/jmxtrans/jmxtrans/archive/jmxtrans-parent-271.tar.gz

influxdb 安装包

# 下载安装包
https://dl.influxdata.com/influxdb/releases/influxdb-1.8.0.x86_64.rpm
# 官网下载页面
https://portal.influxdata.com/downloads/

Grafana 安装包

https://dl.grafana.com/oss/release/grafana-6.7.3-1.x86_64.rpm

安装 influxdb

安装我们刚刚下载 influxdb rpm文件

rpm -ivh influxdb-1.8.0.x86_64.rpm

查看默认配置

> influxd configMerging with configuration at: /etc/influxdb/influxdb.conf
reporting-disabled = false
bind-address = "127.0.0.1:8088"[meta]dir = "/var/lib/influxdb/meta"retention-autocreate = truelogging-enabled = true[data]dir = "/var/lib/influxdb/data"index-version = "inmem"wal-dir = "/var/lib/influxdb/wal"wal-fsync-delay = "0s"validate-keys = falsequery-log-enabled = truecache-max-memory-size = 1073741824cache-snapshot-memory-size = 26214400cache-snapshot-write-cold-duration = "10m0s"compact-full-write-cold-duration = "4h0m0s"compact-throughput = 50331648compact-throughput-burst = 50331648max-series-per-database = 1000000max-values-per-tag = 100000max-concurrent-compactions = 0max-index-log-file-size = 1048576series-id-set-cache-size = 100series-file-max-concurrent-snapshot-compactions = 0trace-logging-enabled = falsetsm-use-madv-willneed = false[coordinator]write-timeout = "10s"max-concurrent-queries = 0query-timeout = "0s"log-queries-after = "0s"max-select-point = 0max-select-series = 0max-select-buckets = 0[retention]enabled = truecheck-interval = "30m0s"[shard-precreation]enabled = truecheck-interval = "10m0s"advance-period = "30m0s"[monitor]store-enabled = truestore-database = "_internal"store-interval = "10s"[subscriber]enabled = truehttp-timeout = "30s"insecure-skip-verify = falseca-certs = ""write-concurrency = 40write-buffer-size = 1000[http]enabled = truebind-address = ":8086"auth-enabled = falselog-enabled = truesuppress-write-log = falsewrite-tracing = falseflux-enabled = falseflux-log-enabled = falsepprof-enabled = truepprof-auth-enabled = falsedebug-pprof-enabled = falseping-auth-enabled = falsehttps-enabled = falsehttps-certificate = "/etc/ssl/influxdb.pem"https-private-key = ""max-row-limit = 0max-connection-limit = 0shared-secret = ""realm = "InfluxDB"unix-socket-enabled = falseunix-socket-permissions = "0777"bind-socket = "/var/run/influxdb.sock"max-body-size = 25000000access-log-path = ""max-concurrent-write-limit = 0max-enqueued-write-limit = 0enqueued-write-timeout = 30000000000[logging]format = "auto"level = "info"suppress-logo = false[[graphite]]enabled = falsebind-address = ":2003"database = "graphite"retention-policy = ""protocol = "tcp"batch-size = 5000batch-pending = 10batch-timeout = "1s"consistency-level = "one"separator = "."udp-read-buffer = 0[[collectd]]enabled = falsebind-address = ":25826"database = "collectd"retention-policy = ""batch-size = 5000batch-pending = 10batch-timeout = "10s"read-buffer = 0typesdb = "/usr/share/collectd/types.db"security-level = "none"auth-file = "/etc/collectd/auth_file"parse-multivalue-plugin = "split"[[opentsdb]]enabled = falsebind-address = ":4242"database = "opentsdb"retention-policy = ""consistency-level = "one"tls-enabled = falsecertificate = "/etc/ssl/influxdb.pem"batch-size = 1000batch-pending = 5batch-timeout = "1s"log-point-errors = true[[udp]]enabled = falsebind-address = ":8089"database = "udp"retention-policy = ""batch-size = 5000batch-pending = 10read-buffer = 0batch-timeout = "1s"precision = ""[continuous_queries]log-enabled = trueenabled = truequery-stats-enabled = falserun-interval = "1s"[tls]min-version = ""max-version = ""

修改参数

默认influxDB使用以下端口

8086: 用于客户端和服务端交互的HTTP API
8088: 用于提供备份和恢复的RPC服务

我这里修改配置文件,使用8087端口，我这里8088和其他服务冲突了

同时修改了数据保存的路径

> vim /etc/influxdb/influxdb.confbind-address = "127.0.0.1:8087"
# metadata 保存路径
dir = "/root/jast/influxdb/meta"
#数据保存路径
dir = "/root/jast/influxdb/data"
#`write-ahead-log(WAL)保存路径
wal-dir = "/root/jast/influxdb/wal"

注意：选择的路径要有权限，否则会启动失败

启动 influxdb

systemctl start influxdb

查看启动状态

systemctl status influxdb

此时influxdb已经启动成功

我们也可以指定配置文件启动，在/etc/influxdb/influxdb.conf ,是默认的目录也可以不指定

influxd -config /etc/influxdb/influxdb.conf

设置基本配置

在服务器输入 influx 进入交互页面

[root@ecs-t-001-0001 influx]# influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
>

创建用户

CREATE USER "admin" WITH PASSWORD '123456' WITH ALL PRIVILEGES

创建数据库（后面保存数据用）

create database "jmxDB"

查看是否创建成功

[root@ecs-t-001-0001 jmxtrans]# influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
> show databases
name: databases
name
----
_internal
jmxDB
>

influxdb 其他命令扩展

#创建数据库
create database "db_name"
#显示所有的数据库
show databases
#删除数据库
drop database "db_name"
#使用数据库
use db_name
#显示该数据库中所有的表
show measurements
#创建表，直接在插入数据的时候指定表名
insert test,host=127.0.0.1,monitor_name=test count=1
#删除表
drop measurement "measurement_name"
#退出
quit

安装 jmxtrans

安装我们刚刚下载jmxtrans rpm文件

rpm -ivh jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm

安装完成后默认安装目录在

[root@ecs-t-001-0001 jmxtrans]# whereis jmxtrans
jmxtrans: /usr/share/jmxtrans

这里我们先简单配置Kafka 的Memory 监控，其他配置在文末统一整理

我们创建json文件供jmxtrans读取，json文件名称自己根据业务取名即可

{"servers" : [ {"port" : "9393","host" : "172.11.0.1","queries" : [ {"obj" : "java.lang:type=Memory","attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],"resultAlias":"jvmMemory","outputWriters" : [ {"@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory","url" : "http://172.11.0.1:8086/","username" : "admin","password" : "123456","database" : "jmxDB","tags"     : {"application" : "kafka_server"}} ]} ]} ]
}

简单解释一下上面的说明

port: 我们要监控的Kafka JMX端口
host：我们要监控的Kafka host
resultAlias：自定义表名，收集到的数据会存入influxdb的定义的表中，自动创建
outputWriters为连接influxdb的配置
@class不需要修改
url：influxdb的机器+端口，默认端口8086
username和password：influxdb的用户和密码
database：influxdb数据库（我们刚刚创建的）

启动之前我们把 /usr/share/jmxtrans 目录下的所有 .json 文件换个名，因为它会默认会读取 /usr/share/jmxtrans 目录下的所有json文件

在 /usr/share/jmxtrans 目录下启动 jmxtrans.sh

jmxtrans.sh start

到这里正常来说就是要启动成功了，我们先说下可能遇到的异常

可能遇到的异常

异常1

[root@ecs-t-001-0001 jmxtrans]# Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=384m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
MaxTenuringThreshold of 16 is invalid; must be between 0 and 15
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

MaxTenuringThreshold 这个参数用于控制对象能经历多少次Minor GC才晋升到旧生代

提示设置的是16，但是范围在0-15，我们直接修改一下启动脚本 jmxtrans.sh

> vim jmxtrans.shGC_OPTS=${GC_OPTS:-"-Xms${HEAP_SIZE}M -Xmx${HEAP_SIZE}M -XX:+UseConcMarkSweepGC -XX:NewRatio=${NEW_RATIO} -XX:NewSize=${NEW_SIZE}m -XX:MaxNewSize=${NEW_SIZE}m -XX:MaxTenuringThreshold=15 -XX:GCTimeRatio=9 -XX:PermSize=${PERM_SIZE}m -XX:MaxPermSize=${MAX_PERM_SIZE}m -XX:+UseTLAB -XX:CMSInitiatingOccupancyFraction=${IO_FRACTION} -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=${CPU_CORES} -Dsun.rmi.dgc.server.gcInterval=28800000 -Dsun.rmi.dgc.client.gcInterval=28800000"}

异常2

[root@ecs-t-001-0001 jmxtrans]# Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=384m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Exception in thread "main" com.googlecode.jmxtrans.util.LifecycleException: com.googlecode.jmxtrans.util.LifecycleException: Error parsing json: /var/lib/jmxtrans/kafka.jsonat com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:146)at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
Caused by: com.googlecode.jmxtrans.util.LifecycleException: Error parsing json: /var/lib/jmxtrans/kafka.jsonat com.googlecode.jmxtrans.JmxTransformer.processFilesIntoServers(JmxTransformer.java:358)at com.googlecode.jmxtrans.JmxTransformer.startupSystem(JmxTransformer.java:301)at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:142)... 2 more
Caused by: java.lang.IllegalArgumentException: Invalid type id 'com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory' (for id type 'Id.class'): no such class foundat org.codehaus.jackson.map.jsontype.impl.ClassNameIdResolver.typeFromId(ClassNameIdResolver.java:89)at org.codehaus.jackson.map.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:73)at org.codehaus.jackson.map.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:65)at org.codehaus.jackson.map.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:81)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:118)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:93)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:149)at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:237)at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:496)at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:350)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:116)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:93)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:149)at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:237)at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:496)at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:350)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:116)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:93)at org.codehaus.jackson.map.deser.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)at org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:149)at org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:237)at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:496)at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:350)at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:1980)at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1225)at com.googlecode.jmxtrans.util.JmxUtils.getJmxProcess(JmxUtils.java:494)at com.googlecode.jmxtrans.JmxTransformer.processFilesIntoServers(JmxTransformer.java:352)... 4 more

说是解析 com.googlecode.jmxtrans.util.LifecycleException 异常，这里需要我们自己编译一下jar包，在文章开头我们下载过jmxtrans源码

在项目目录下进行编译，文末有我编译好的jar包

mvn clean package -Dmaven.test.skip=true -DskipTests=true;

编译完成我们需要的jar包在 jmxtrans-jmxtrans-parent-271\jmxtrans\target 目录下

jmxtrans-271-all.jar 就是我们需要用的jar包

将jar包传到jmxtrans目录下

我们对比一下发现我们编译的包是有这个类的，而他自带的那个没有

[root@ecs-t-001-0001 jmxtrans]# grep 'com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory' ./jmxtrans-271-all.jar
Binary file ./jmxtrans-271-all.jar matches
[root@ecs-t-001-0001 jmxtrans]# grep 'com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory' ./jmxtrans-all.jar
[root@ecs-t-001-0001 jmxtrans]#

替换 jmxtrans.sh 中应用的 jmxtrans jar包名称

#JAR_FILE=${JAR_FILE:-"jmxtrans-all.jar"}
JAR_FILE=${JAR_FILE:-"jmxtrans-271-all.jar"}

再次启动即可

验证jmxtrans是否成功运行

进入 influx jmxDB数据库(我们之前创建的)，查看表 show MEASUREMENTS，我们发现自动创了jvmMemory

[root@ecs-t-001-0001 jmxtrans]# influx
Connected to http://localhost:8086 version 1.8.0
InfluxDB shell version: 1.8.0
> show databases
name: databases
name
----
_internal
jmxDB
> use jmxDB
Using database jmxDB
> show
ERR: error parsing query: found EOF, expected CONTINUOUS, DATABASES, DIAGNOSTICS, FIELD, GRANTS, MEASUREMENT, MEASUREMENTS, QUERIES, RETENTION, SERIES, SHARD, SHARDS, STATS, SUBSCRIPTIONS, TAG, USERS at line 1, char 6
> show MEASUREMENTS
name: measurements
name
----
jvmMemory

具体查看数据，发现有数据写入

至此 jmxtrans 已成功监控 Kafka JMX端口，离成功更近了

安装 Grafana

安装

yum install grafana-6.7.3-1.x86_64.rpm

配置文件默认路径 /etc/grafana/grafana.ini

修改下web端口

> vim /etc/grafana/grafana.ini# web页面端口默认3000
http_port = 9099

启动服务并设置为开机启动

systemctl start grafana-server
systemctl enable grafana-server

查看启动状态

systemctl  status grafana-server

访问 web页面，第一次登陆账号密码是 admin/admin ，登陆完成后会提示你设置密码

开始配置Grafana显示模板

点击DataSource

选择添加 InfluxDB ，并填写基本信息

填写 influxDB 数据库信息

点击保存

此时InfluxDB配置完成，我们几区创建仪表盘

选择添加一个查询

选择我们上面设置的KafkaMonitor（因为我们只有一个默认的也是这个）

简单修改一下sql，因为我们上面只监控了JMX中的内存信息

注意：如果上面使用tag进行划分，这里就不要设置了，否则都叫Jvm内存使用了

简单配置完成，我们对比一下监控与jconsole监控对比

influxDB 与 Grafana 监控模板

模板下载

链接：https://pan.baidu.com/s/1ld-Yhv7wVutRxslV084GoQ
提取码：0pzr
复制这段内容后打开百度网盘手机App，操作更方便哦

Kafka JMX 监控之 jmxtrans + influxdb + grafana相关推荐

Docker容器监控可视化cAdivisor+InfluxDB+Grafana
问题在日常的Docker使用过程中, 要想了解容器当前的运行状况 , 通过docker stats命令就可以很方便的看到当前宿主机上所有容器的CPU,内存以及网络流量等数据 , 但是, docke ...
docker swarm集群监控方案cAdvisor+InfluxDB+Grafana实战
docker swarm集群的监控方案很多,cAdvisor+InfluxDB+Grafana方案功能强大灵活.最重要的是这个方案开源.免费.易用,是不花钱版监控方案.参考文档:https://bot ...
Kafka JMX监控报错 Failed to get broker metrics for BrokerIdentity(128,192.168.2.128,9999,true,false,Map
KafkaManager报错 2019-05-19 14:21:53,817 - [ERROR] k.m.a.c.BrokerViewCacheActor - Failed to get broker ...
【kafka】Kafka JMX监控报错 Failed to get broker metrics for BrokerIdentity
1.背景 KafkaManager报错 2019-05-19 14:21:53,817 - [ERROR] k.m.a
influxdb+grafana 监控
参考InfluxDB 入门一.简介 1.什么是InfluxDB InfluxDB(时序数据库),常用的一种使用场景:监控数据统计.每毫秒记录一下电脑内存的使用情况,然后就可以根据统计的数据,利用图形 ...
Kafka常用监控框架百科全书
Kafka搭建好投入使用后,为了运维更便捷,借助一些管理工具很有必要.Kafka社区似乎一直没有在监控框架方面投入太多的精力,目前Kafka监控方案看似很多,然而并没有一个"大而全" ...
基于InfluxDB+Grafana打造大数据监控利器--转
这是一个大数据爆发的时代.面对信息的激流.多元化数据的涌现,我们在获取.存储.传输.理解.分析.应用.维护大数据时,无疑需要一种便捷的信息交流通道,以便快速.有效.准确地理解和驾驭这个过程.本文将通过 ...
kafka metric java,jmx_exporter+prometheus+grafana实现kafka metric监控
2.配置kafka的JMX PORT,修改bin/kafka-server-start.sh,添加一行export JMX_PORT="9999",我这里用了9999端口. 3.下 ...
基于InfluxDB+Grafana打造大数据监控利器
转载:https://www.cnblogs.com/davidwang456/p/7795263.html 这是一个大数据爆发的时代.面对信息的激流.多元化数据的涌现,我们在获取.存储.传输.理解. ...

Kafka JMX 监控之 jmxtrans + influxdb + grafana

效果图

环境准备

安装 influxdb

安装我们刚刚下载 influxdb rpm文件

查看默认配置

修改参数

启动 influxdb

查看启动状态

设置基本配置

influxdb 其他命令扩展

安装 jmxtrans

可能遇到的异常

验证jmxtrans是否成功运行

安装 Grafana

安装

influxDB 与 Grafana 监控模板

Kafka JMX 监控之 jmxtrans + influxdb + grafana相关推荐

最新文章

热门文章

Kafka JMX 监控 之 jmxtrans + influxdb + grafana

效果图

环境准备

安装 influxdb

安装我们刚刚下载 influxdb rpm文件

查看默认配置

修改参数

启动 influxdb

查看启动状态

设置基本配置

influxdb 其他命令扩展

安装 jmxtrans

可能遇到的异常

验证jmxtrans是否成功运行

安装 Grafana

安装

influxDB 与 Grafana 监控模板

Kafka JMX 监控 之 jmxtrans + influxdb + grafana相关推荐

最新文章

热门文章

Kafka JMX 监控之 jmxtrans + influxdb + grafana

Kafka JMX 监控之 jmxtrans + influxdb + grafana相关推荐