容器云系列之Docker容器监控cAdvisor

容器监控工具很多，本文主要对比了cAdvisor、Weave Scope和Prometheus几种监控工具的特性和功能，并结合环境部署测试了各工具的使用。

容器监控工具很多，下表对比了cAdvisor、Weave Scope和Prometheus等几种监控工具在部署便捷性、信息详细度、集成度、告警功能、非Docker资源监控以及软件成本等方面的特性和功能：

1、Docker Stats命令

单机Docker容器的监控可以使用Docker stats命令，运行docker stats [CONTAINER_NAME]可以查看容器统计信息，包括每个容器的CPU利用率、内存的使用量以及可用内存总量，也可以看到容器通过网络发送和接收的数据总量。如果没有限制容器内存，那么该命令将显示主机的内存总量，但并不意味着你的每个容器都能访问那么多的内存。

[root@tango-01 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
22c64450cb02        ubuntu              "/bin/bash"         3 weeks ago         Up 2 seconds                            test2
[root@tango-01 ~]# docker stats test2
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
22c64450cb02        test2               0.00%               548KiB / 1.777GiB   0.03%               2.23kB / 0B         8.3MB / 0B          1

2、cAdvisor

Docker stats命令和远程API来获取容器的状态信息，但是反映的是当前值，不能看到变化趋势。谷歌提供的图形化工具cAdvisor可以显示docker stats命令获取的数据的可视化界面。cAdvisor部署为一个运行中的daemon，它会收集、聚集、处理并导出运行中容器的信息。这些信息能够包含容器级别的资源隔离参数、资源的历史使用状况、反映资源使用和网络统计数据完整历史状况的柱状图。

运行以下Docker命令安装cAdvisor容器：

[root@tango-01 ~]# docker run   --volume=/:/rootfs:ro   --volume=/var/run:/var/run:rw    --volume=/sys:/sys:ro   --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080   --detach=true  --name=cadvisor   google/cadvisor:latest
Unable to find image 'google/cadvisor:latest' locally
latest: Pulling from google/cadvisor
ff3a5c916c92: Pull complete
44a45bb65cdf: Pull complete
0bbe1a2fe2a6: Pull complete
Digest: sha256:815386ebbe9a3490f38785ab11bda34ec8dacf4634af77b8912832d4f85dca04
Status: Downloaded newer image for google/cadvisor:latest
b25ac9c116e79b7975bfb6dad93f03fff9c8e1db58d63253aac3f0bd8755ad7e
[root@tango-01 ~]# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                    NAMES
b25ac9c116e7        google/cadvisor:latest   "/usr/bin/cadvisor -…"   10 seconds ago      Up 8 seconds        0.0.0.0:8080->8080/tcp   cadvisor
22c64450cb02        ubuntu                   "/bin/bash"              3 weeks ago         Up 5 minutes                                 test2

访问http://192.168.112.10:8080/可以看到cAdvisor的界面，可以看到CPU的使用率、内存使用率、网络吞吐量以及磁盘空间利用率等信息。

3、cAdvisor+Influxdb+grafana实现跨主机容器监控

如果要实现跨主机容器监控，最经典的方案是cAdvisor+Influxdb+grafana，在每台主机上运行一个cAdvisor容器负责数据采集，再将采集后的数据存储到时序数据库Influxdb，再通过图形展示工具grafana定制展示画面。具体结构如下：

1）在tango-01安装并启动Influxdb

[root@tango-01 ~]# docker run -itd -p 8083:8083 -p 8086:8086  --name influxdb tutum/influxdb
Unable to find image 'tutum/influxdb:latest' locally
latest: Pulling from tutum/influxdb
a3ed95caeb02: Pull complete
23efb549476f: Pull complete
aa2f8df21433: Pull complete
ef072d3c9b41: Pull complete
c9f371853f28: Pull complete
a248b0871c3c: Pull complete
749db6d368d0: Pull complete
db2492acfcc3: Pull complete
b7e7d2e12d53: Pull complete
4272a53eef10: Pull complete
9b2fefdb5321: Pull complete
Digest: sha256:5b7c5e318303ad059f3d1a73d084c12cb39ae4f35f7391b79b0ff2c0ba45304b
Status: Downloaded newer image for tutum/influxdb:latest
2cb9da6f101e7f9ac2556464825ec5b95520a5e22510c8d02c79b84b9442a100
[root@tango-01 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                            NAMES
2cb9da6f101e        tutum/influxdb      "/run.sh"           7 seconds ago       Up 4 seconds        0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp   influxdb
[root@tango-01 ~]#

通过http://192.168.112.10:8083/访问控制台

2）进入Influxdb容器安装cAdvisor数据库

[root@tango-01 ~]# docker exec -it influxdb /bin/bash
root@2cb9da6f101e:/# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 1.0.0
InfluxDB shell version: 1.0.0
> create database cadvisor
> use cadvisor
Using database cadvisor
> create user "root" with password 'root' with all privileges

3）在tango-centos01和tango-centos02启动cadvisor容器

[root@tango-centos01 ~]# docker run -itd --name cadvisor --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker:/var/lib/docker:ro -p 8080:8080 google/cadvisor:latest -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_user=root -storage_driver_password=root -storage_driver_host=192.168.112.10:8086
feeb282b3d48d8d64b50ee5a995cfa3271087ecd5d3ce1679f1a406220b992b0
[root@tango-centos01 ~]# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                    NAMES
feeb282b3d48        google/cadvisor:latest   "/usr/bin/cadvisor -â€¦"   7 seconds ago       Up 4 seconds        0.0.0.0:8080->8080/tcp   cadvisor
[root@tango-centos02 ~]# docker run -itd --name cadvisor --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker:/var/lib/docker:ro -p 8080:8080 google/cadvisor:latest -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_user=root -storage_driver_password=root -storage_driver_host=192.168.112.10:8086
Unable to find image 'google/cadvisor:latest' locally
latest: Pulling from google/cadvisor
ff3a5c916c92: Pull complete
44a45bb65cdf: Pull complete
0bbe1a2fe2a6: Pull complete
Digest: sha256:815386ebbe9a3490f38785ab11bda34ec8dacf4634af77b8912832d4f85dca04
Status: Downloaded newer image for google/cadvisor:latest
c0e1b802e19e82c680a3534c1e595e184eb12e6207302bfa00522a863f010617
[root@tango-centos02 ~]# docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                    NAMES
c0e1b802e19e        google/cadvisor:latest   "/usr/bin/cadvisor -â€¦"   15 seconds ago      Up 12 seconds       0.0.0.0:8080->8080/tcp   cadvisor

通过访问控制台http://192.168.112.101:8080/，可以看到在 Docker 主机和容器上收集统计信息的 cAdvisor

输入命令SHOW MEASUMERENTS可以看到信息说明采集成功

4）在tango-01启动grafana容器

[root@tango-01 ~]# docker run -itd -p 3000:3000 -e INFLUXDB_HOST=192.168.112.10 -e INFLUXDB_PORT=8086 -e INFLUDB_NAME=cadvisor -e INFLUXDB_USER=root -e INFLUXDB_PASS=root --name grafana grafana/grafana
Unable to find image 'grafana/grafana:latest' locally
latest: Pulling from grafana/grafana
188c0c94c7c5: Pull complete
94db9aec2a58: Pull complete
618563756337: Pull complete
f413b64712e1: Pull complete
18534279664f: Pull complete
4f4fb700ef54: Pull complete
3abba839cfd4: Pull complete
aac362f49dd4: Pull complete
Digest: sha256:9f43d0ac1fdecdd08a47bce4038fa5c9c67cc84fc025df78b87ae7b7b076aee9
Status: Downloaded newer image for grafana/grafana:latest
2d9fafe3a2c5b453da5ff169e0e32ba4a04799b8a2c746cd66ee812a0f45f3d8
[root@tango-01 ~]# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                                            NAMES
2d9fafe3a2c5        grafana/grafana     "/run.sh"           5 seconds ago       Up 3 seconds        0.0.0.0:3000->3000/tcp                           grafana
2cb9da6f101e        tutum/influxdb      "/run.sh"           34 minutes ago      Up 34 minutes       0.0.0.0:8083->8083/tcp, 0.0.0.0:8086->8086/tcp   influxdb

访问控制台登录到grafana，用户密码为admin/admin

5）配置grafana

成功登入后，配置数据源

在 Data Source Settings 界面填写相关信息

当点击 Save&Test 返回 Data source is working 信息的时候，表示 grafana 已经正确连接到相应的容器

参考资料：

https://docs.docker.com/engine/reference/commandline/stats/
https://github.com/google/cadvisor
https://www.cnblogs.com/itzhao/p/11368753.html

转载请注明原文地址：https://blog.csdn.net/solihawk/article/details/121624212
文章会同步在公众号“牧羊人的方向”更新，感兴趣的可以关注公众号，谢谢！