Nomad 多job/group/task调度测试

关系：job 包含 group 包含 task

1 搭建nomad集群

本测试使用三台ubuntu18.04虚拟机，IP地址分别为：

虚拟机1：192.168.60.10
虚拟机2：192.168.60.11
虚拟机3：192.168.60.12

具体搭建方法见Nomad集群自身高可用测试。

2 测试driver=docker

使用的docker镜像为nginx的本地拷贝镜像nginx:v1，见Nomad Nginx 暴露IP端口和重启/重调度验证。

2.1 多job测试

将以下job同时执行四次nomad job run nginx[1-4].nomad，注意每次都要将job、group、task名字修改：

job "nginx1" {datacenters = ["dc1"]type = "service"group "nginxg1" {count = 1network {port "nginxport" {static = 8765to = 80}}task "nginxt1" {driver = "docker"config {image = "nginx:v1"ports = ["nginxport"]}}}
}

static = 8765这个端口号需要不同，否则肯定会分配到不同的虚拟机上，避免端口被占用。

再执行nomad job status如下：

ubuntu1@ubuntu1$ nomad job status
ID      Type     Priority  Status   Submit Date
nginx1  service  50        running  2021-08-30T13:35:02+08:00
nginx2  service  50        running  2021-08-30T13:35:16+08:00
nginx3  service  50        running  2021-08-30T13:36:17+08:00
nginx4  service  50        running  2021-08-30T13:36:34+08:00

在每个虚拟机中docker ps查看结果，虚拟机2运行了4个nginx。

如果我在第一个job里规定只能在虚拟机1运行，后面三个也都会在虚拟机1运行。

这个机制我也不太明白，可能是这些job还不够占资源。

停止job：

nomad job stop nginx1
nomad job stop nginx2
nomad job stop nginx3
nomad job stop nginx4

2.2 多group测试

执行以下job文件nomad job run nginx.nomad，1个job包含了4个group，每个group有1个task：

job "nginx" {datacenters = ["dc1"]group "nginxg1" {count = 1network {port "nginxport" {static = 8765to = 80}}task "nginxt1" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport"]}}}group "nginxg2" {count = 1network {port "nginxport" {static = 8764to = 80}}task "nginxt2" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport"]}}}group "nginxg3" {count = 1network {port "nginxport" {static = 8763to = 80}}task "nginxt3" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport"]}}}group "nginxg4" {count = 1network {port "nginxport" {static = 8762to = 80}}task "nginxt4" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport"]}}}
}

结果与2.1 多job测试相同。

停止job：nomad job stop nginx

2.3 多task测试

执行以下job文件nomad job run nginx.nomad，1个job包含了1个group，每个group有4个task：

job "nginx" {datacenters = ["home"]group "nginxg" {count = 1network {port "nginxport1" {static = 8765to = 80}port "nginxport2" {static = 8764to = 80}port "nginxport3" {static = 8763to = 80}port "nginxport4" {static = 8762to = 80}}task "nginxt1" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport1"]}}task "nginxt2" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport2"]}}task "nginxt3" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport3"]}}task "nginxt4" {driver = "docker"config {image = "nginx:v2"ports = ["nginxport4"]}}}
}

运行后发现，虚拟机2上运行了4个nginx：

ubuntu2@ubuntu2:~$ sudo docker ps
CONTAINER ID   IMAGE      COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
4be8151eaf2a   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8762->80/tcp, 192.168.60.11:8762->80/udp   nginxt4-bd4038a2-a662-eb24-e8e5-f83906a2ece6
b090014faf2e   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8764->80/tcp, 192.168.60.11:8764->80/udp   nginxt2-bd4038a2-a662-eb24-e8e5-f83906a2ece6
fc3be23df0ea   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8765->80/tcp, 192.168.60.11:8765->80/udp   nginxt1-bd4038a2-a662-eb24-e8e5-f83906a2ece6
b2670ffe0f10   nginx:v1   "/docker-entrypoint.…"   20 seconds ago   Up 18 seconds   192.168.60.11:8763->80/tcp, 192.168.60.11:8763->80/udp   nginxt3-bd4038a2-a662-eb24-e8e5-f83906a2ece6

说明在driver=docker（type=service）时，1个group内的所有task都会被分配到同一个client上。

3 测试driver=raw_exec

编写简单程序main.c：

#include <stdio.h>
int main()
{while(1){sleep(10);}return 0;
}

gcc main.c -o main.out编译。

sudo cp main.out /usr/将main.out拷贝到/usr下面。

在/etc/nomad.d/nomad.hcl文件中添加以下内容：

plugin "raw_exec" {config {enabled = true}
}

三个虚拟机都要完成以上操作。

3.1 多job测试

将以下job同时执行4次nomad job run exec[1-4].nomad，注意每次都要将job、group、task名字修改：

job "exec1" {datacenters = ["dc1"]type = "batch"group "execg1" {count = 1task "exect1" {driver = "raw_exec"config {command = "/usr/main.out"}}}
}

再执行nomad job status如下：

ubuntu1@ubuntu1$ nomad job status
ID      Type     Priority  Status          Submit Date
exec1   batch    50        running         2021-08-31T11:11:09+08:00
exec2   batch    50        running         2021-08-31T11:11:13+08:00
exec3   batch    50        running         2021-08-31T11:11:16+08:00
exec4   batch    50        running         2021-08-31T11:11:19+08:00

发现在ubuntu2和ubuntu3下分别都运行了2个main.out进程，ubuntu1没有。

说明在driver=raw_exec（type=batch）时，1个client可以运行多个job，且并不会优先分配给没有job的client。

停止job：

nomad job stop exec1
nomad job stop exec2
nomad job stop exec3
nomad job stop exec4

3.2 多group测试

执行以下job文件nomad job run exec.nomad，1个job包含了4个group，每个group有1个task：

job "exec" {datacenters = ["dc1"]type = "batch"group "execg1" {count = 1task "exect1" {driver = "raw_exec"config {command = "/usr/main.out"}}}group "execg2" {count = 1task "exect2" {driver = "raw_exec"config {command = "/usr/main.out"}}}group "execg3" {count = 1task "exect3" {driver = "raw_exec"config {command = "/usr/main.out"}}}group "execg4" {count = 1task "exect4" {driver = "raw_exec"config {command = "/usr/main.out"}}}
}

也可以只有1个group，但是count=4，效果相同

结果ubuntu1运行了1个main.out进程，ubuntu3运行了3个，ubuntu2没有。

说明在driver=raw_exec（type=batch）时，1个client可以运行多个group，且并不会优先分配给没有group的client。

停止job：nomad job stop exec

3.3 多task测试

执行以下job文件nomad job run exec.nomad，1个job包含了1个group，每个group有4个task：

job "exec" {datacenters = ["dc1"]type = "batch"group "execg" {count = 1task "exect1" {driver = "raw_exec"config {command = "/usr/main.out"}}task "exect2" {driver = "raw_exec"config {command = "/usr/main.out"}}task "exect3" {driver = "raw_exec"config {command = "/usr/main.out"}}task "exect4" {driver = "raw_exec"config {command = "/usr/main.out"}}}
}

结果ubuntu1上运行了4个main.out进程，2和3没有。

说明1个group内的所有task都会被分配到同一个client上。

3.4 小结

driver=raw_exec（type=batch）时：

group是运行任务的基本单位（task是最小单位）
每个client(node)可以执行多个group
同属于一个group的task，都会在同一个client运行