docker（11、Docker Swarm4）11、副本数量（replicated 和 global ）12、Label 控制 Service 的位置 13、如何配置 Health Check

11、副本数量模式（replicated mode 和 global mode）

Swarm 可以在 Service 创建和运行过程中灵活的通过 --replicas 调整容器副本数量，内部调度器则会根据当前集群的资源使用情况在不同的node 上启停容器，这就是 Service 默认的 replicated mode 。在此模式下，node 上运行的副本数有多有少，一般情况下，资源更丰富的node 运行的副本数更多，反之亦然。

除了 replicated mode ，Service 还提供了一个 global mode ，其作用是强制在每个node上都运行一个且最多一个副本。

global mode 特别适合需要运行daemon 的集群环境。比如要收集所有容器的日志，就可以用 global mode 创建 Service，在所有 node 上都运行 gliderlabs/logspout 容器，及时之后有新的node加入，swarm 也会自动在新的node上启动一个 gliderlabs/logspout 副本

[root@swarm-manager ~]# docker service create --mode global --name bbox busybox sleep 99999
5qamycgjpsivklii7omh4wgq8
overall progress: 2 out of 2 tasks
l9usnqu08wnq: running   [==================================================>]
su6as9kbp9od: running   [==================================================>]
verify: Service converged
[root@swarm-manager ~]#
[root@swarm-manager ~]# docker service ps bbox
ID                  NAME                             IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ygwk31ee5cek        bbox.su6as9kbp9odmbemdasjgvsd1   busybox:latest      swarm-worker2       Running             Running 40 seconds ago
kswnk367tpyu        bbox.l9usnqu08wnqrejs6v3a3c9j7   busybox:latest      swarm-worker1       Running             Running 40 seconds ago
[root@swarm-manager ~]#

例如swarm-manager是Drain 状态通过docker node update swarm-manager --availability active GLobal会在swarm-manager创建副本，global模式会在每台机器上创建副本

可以通过 docker service inspect 查看 service 的 mode。

[root@swarm-manager ~]# docker service inspect bbox  --prettyID:        5qamycgjpsivklii7omh4wgq8
Name:       bbox
Service Mode:   Global
Placement:
UpdateConfig:Parallelism:   1On failure:    pauseMonitoring Period: 5sMax failure ratio: 0Update order:      stop-first
RollbackConfig:Parallelism: 1On failure:    pauseMonitoring Period: 5sMax failure ratio: 0Rollback order:    stop-first
ContainerSpec:Image:        busybox:latest@sha256:836945da1f3afe2cfff376d379852bbb82e0237cb2925d53a13f53d6e8a8c48cArgs:        sleep 99999 Init:       false
Resources:
Endpoint Mode:  vip[root@swarm-manager ~]#

Service Mode: Global

这里是 Global，如果创建 service 时不指定，默认是 Replicated。

12、Label 控制 Service 的位置

使用 label能精细控制 Service 的运行位置呢

逻辑分两步：

1、为每个 node 定义 label。
2、设置 service 运行在指定 label 的 node 上。label 可以灵活描述 node 的属性，其形式是 key=value，用户可以任意指定，
例如将 swarm-worker1 作为测试环境，为其添加 label env=test：
swarm-worker1标记为测试环境，swarm-worker2标记为线上环境

swarm-worker1

[root@swarm-manager ~]# docker node update --label-add env=test swarm-worker1
swarm-worker1
[root@swarm-manager ~]# docker node inspect swarm-worker1 --pretty
ID:         l9usnqu08wnqrejs6v3a3c9j7
Labels:- env=test
Hostname:               swarm-worker1
Joined at:              2020-05-12 06:57:51.232356307 +0000 utc
Status:State:           ReadyAvailability:          Active

swarm-worker2

[root@swarm-manager ~]# docker node update --label-add env=prod swarm-worker2
swarm-worker2
[root@swarm-manager ~]# docker node inspect swarm-worker2 --pretty
ID:         su6as9kbp9odmbemdasjgvsd1
Labels:- env=prod
Hostname:               swarm-worker2
Joined at:              2020-05-12 06:57:54.918523002 +0000 utc
Status:State:           ReadyAvailability:          Active

在测试环境创建Service my_web ，副本数 3

[root@swarm-manager ~]# docker service create --name my_web --replicas 3 --constraint node.labels.env==test --publish 80:80 httpd
3yafgv7g6xhs7zi89evu4rrct
overall progress: 3 out of 3 tasks
1/3: running   [==================================================>]
2/3: running   [==================================================>]
3/3: running   [==================================================>]
verify: Service converged
[root@swarm-manager ~]# docker service ps my_web
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
sdax5qedqp56        my_web.1            httpd:latest        swarm-worker1       Running             Running 2 minutes ago
kjnt4sx2tuu4        my_web.2            httpd:latest        swarm-worker1       Running             Running 2 minutes ago
ddjygvz1wz8h        my_web.3            httpd:latest        swarm-worker1       Running             Running 2 minutes ago
[root@swarm-manager ~]# docker service inspect my_web --prettyID:      3yafgv7g6xhs7zi89evu4rrct
Name:       my_web
Service Mode:   ReplicatedReplicas: 3
Placement:Constraints:  [node.labels.env==test]

确认所有的副本都在测试环境的swarm-worker1上可以通过 docker service inspect 查看 --constraint 的设置：

 Constraints:    [node.labels.env==test]

将 my_web 的Service 从测试环境迁移到线上环境

① 删除Service 测试环境的标签

[root@swarm-manager ~]# docker service update --constraint-rm node.labels.env==test my_web
my_web
overall progress: 3 out of 3 tasks
1/3: running   [==================================================>]
2/3: running   [==================================================>]
3/3: running   [==================================================>]
verify: Service converged
[root@swarm-manager ~]# docker service ps my_web
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
sdax5qedqp56        my_web.1            httpd:latest        swarm-worker1       Running             Running 8 minutes ago
kjnt4sx2tuu4        my_web.2            httpd:latest        swarm-worker1       Running             Running 8 minutes ago
ddjygvz1wz8h        my_web.3            httpd:latest        swarm-worker1       Running             Running 8 minutes ago
[root@swarm-manager ~]#

②添加Service 线上环境的标签

[root@swarm-manager ~]# docker service update --constraint-add node.labels.env==prod my_web
my_web
overall progress: 3 out of 3 tasks
1/3: running   [==================================================>]
2/3: running   [==================================================>]
3/3: running   [==================================================>]
verify: Service converged
[root@swarm-manager ~]#

③查看服务

[root@swarm-manager ~]# docker service ps my_web
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE                 ERROR               PORTS
04j462joi628        my_web.1            httpd:latest        swarm-worker2       Running             Running about a minute ago
sdax5qedqp56         \_ my_web.1        httpd:latest        swarm-worker1       Shutdown            Shutdown about a minute ago
pw9l21r7i74n        my_web.2            httpd:latest        swarm-worker2       Running             Running about a minute ago
kjnt4sx2tuu4         \_ my_web.2        httpd:latest        swarm-worker1       Shutdown            Shutdown about a minute ago
p3u8actunqgc        my_web.3            httpd:latest        swarm-worker2       Running             Running about a minute ago
ddjygvz1wz8h         \_ my_web.3        httpd:latest        swarm-worker1       Shutdown            Shutdown about a minute ago
[root@swarm-manager ~]#

label 与 global mode 组合，只收集线上环境的日志

[root@swarm-manager ~]#  docker service create --mode global --constraint node.labels.env==prod --name monitor busybox sleep 99999
hkztjusq6epfhbu4vpsvdvwq0
overall progress: 1 out of 1 tasks
su6as9kbp9od: running   [==================================================>]
verify: Service converged
[root@swarm-manager ~]#
[root@swarm-manager ~]# docker service ps monitor
ID                  NAME                                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
r36jits6gwa5        monitor.su6as9kbp9odmbemdasjgvsd1   busybox:latest      swarm-worker2       Running             Running 52 seconds ago
[root@swarm-manager ~]#

只有 swarm-worker2 节点上才会运行monitor

13、如何配置 Health Check

容器状态是UP的，那应用就是健康的吗？不一定
Docker 只能从容器启动进程的返回代码判断其状态，而对于容器内部应用的运行情况基本没有了解。
执行 docker run 命令时，通常会根据Dockerfile中的CMD或 ENTRYPOINT 启动一个进程，这个进程的状态就是 docker ps STATUS 列显示容器的状态。

上面命令显示：
1、有的容器正在运行，状态为 UP
2、有的容器已经正常停止了，状态是exited(0)
3、有的则因发生故障停止了，退出代码非 0 ，例如 exited (137)、exited(1) 等
即使容器状态是 UP ，也不能保证应用没有问题，web server 虽然没有崩溃，但如果总是返回 HTTP500 - Internal Server Error ，对于应用来说就是严重的故障。
如何从应用的业务层面检查容器状态呢，那就是 Health Check。
Docker 支持的 Health Check 可以是任何一个单独的命令，Docker 会在容器中执行该命令，如果返回 0 ，容器被认为是healthy ，如果返回 1 则为unhealthy。
对于提供HTTP 服务接口的应用，常用的 Health Check 是通过curl 检查HTTP状态码，比如

curl --fail http://localhost:8080 || exit 1

如果 curl 命令检测到任何一个错误的HTTP 状态码，则返回 1 ，Health Check 失败。

[root@swarm-manager ~]# docker service create --name web_server --health-cmd "curl http://localhost:8091/pools || exit 1"  couchbase
p9ee1f9g0yl8n1havcvhjgwur
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged
[root@swarm-manager ~]#  docker service ps web_server
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
88umasee1u6w        web_server.1        couchbase:latest    swarm-worker1       Running             Running 21 minutes ago
[root@swarm-manager ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@swarm-manager ~]#

[root@swarm-worker1 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS                                                        NAMES
26a8db5672a2        couchbase:latest    "/entrypoint.sh couc…"   21 minutes ago      Up 21 minutes (healthy)     8091-8096/tcp, 11207/tcp, 11210-11211/tcp, 18091-18096/tcp   web_server.1.88umasee1u6wqem1vvbrx0wiy

SATUS栏可以看到容器的监控状态healthy

--health-timeout 命令超时时间，默认30s

--health-interval 命令执行间隔，默认30s

--health-retries 命令失败重试的次数，默认为3，如果 3 次都失败了则会将容器标记为unhealthy。swarm会销毁并重建unhealthy的副本。

下面模拟一个unhealthy的场景，让curl 访问一个不存在的urlroot@host03:~# docker service create --name ng_server --health-cmd "curl --fail  http://localhost:8091/non-exist || exit 1" couchbase
y4qappjx5ke1ffp8ltsivt892
overall progress: 0 out of 1 tasks
1/1: starting  [============================================>      ]    #    创建Service一直卡在这里root@host03:~# docker service ps ng_server    过一段时间查看该Service对应的容器，发现有三个处于shutdown状态，这就对应了前面说的，当健康检查不通过的时候会销毁并重建新的副本（容器），这些容器其实的node上只是被shutdown了
ID                  NAME                IMAGE               NODE                DESIRED STATE       CURRENT STATE                 ERROR               PORTS
l0o7fyf6v8to        ng_server.1         couchbase:latest    host02              Ready               Ready 3 seconds ago
j3r6u80e9pe8         \_ ng_server.1     couchbase:latest    host02              Shutdown            Complete 3 seconds ago
177fik81ja20         \_ ng_server.1     couchbase:latest    host02              Shutdown            Complete about a minute ago
4ko4dmanao2y         \_ ng_server.1     couchbase:latest    host02              Shutdown            Complete 3 minutes ago                            下面是在node上查看容器状态，可见累计创建过四个容器（副本），有问题的副本（容器）都shutdown了root@host02:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                                 PORTS                                                        NAMES
0316bb763a73        couchbase:latest    "/entrypoint.sh couc…"   About a minute ago   Up About a minute (health: starting)   8091-8096/tcp, 11207/tcp, 11210-11211/tcp, 18091-18096/tcp   ng_server.1.4ko4dmanao2yntzkati7rqf2vroot@host02:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                                 PORTS                                                        NAMES
c08f7f88d929        couchbase:latest    "/entrypoint.sh couc…"   About a minute ago   Up About a minute (health: starting)   8091-8096/tcp, 11207/tcp, 11210-11211/tcp, 18091-18096/tcp   ng_server.1.177fik81ja20jwr36dz56sfh3root@host02:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                             PORTS                                                        NAMES
eabec2359233        couchbase:latest    "/entrypoint.sh couc…"   24 seconds ago      Up 18 seconds (health: starting)   8091-8096/tcp, 11207/tcp, 11210-11211/tcp, 18091-18096/tcp   ng_server.1.j3r6u80e9pe8101xvt6mixvxcroot@host02:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                            PORTS                                                        NAMES
d7ad2c993564        couchbase:latest    "/entrypoint.sh couc…"   10 seconds ago      Up 4 seconds (health: starting)   8091-8096/tcp, 11207/tcp, 11210-11211/tcp, 18091-18096/tcp   ng_server.1.l0o7fyf6v8tocy60kdsb7djcyroot@host02:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                                 PORTS                                                        NAMES
f13e80a53400        couchbase:latest    "/entrypoint.sh couc…"   1 minutes ago        Up About 1 minutes (health: starting)   8091-8096/tcp, 11207/tcp, 11210-11211/tcp, 18091-18096/tcp  ng_server.1.koawuqt4rjyo65i3kilj18hsi
fe5c0436aaed        couchbase:latest    "/entrypoint.sh couc…"   4 minutes ago        Exited (0) 2 minutes ago                                                                            ng_server.1.nsy7l5anl0megyjfms3nbooyy
7d99a8eb3268        couchbase:latest    "/entrypoint.sh couc…"   5 minutes ago        Exited (0) 4 minutes ago                                                                            ng_server.1.m93f6tpx0hqd2z9sl0jadi8ib
d7ad2c993564        couchbase:latest    "/entrypoint.sh couc…"   7 minutes ago        Exited (0) 5 minutes ago                                                                            ng_server.1.l0o7fyf6v8tocy60kdsb7djcy

Docker 默认只能通过容器进程的返回码判断容器的状态，Health Check 则能够从业务角度判断应用是否发生故障，是否需要重启。