多机Nomad+Consul+consul-template+Nginx反向代理

虚拟机1：Ubuntu18.04，IP：192.168.10.11
虚拟机2：Ubuntu18.04，IP：192.168.10.22

测试过程：Nomad起2个job容器分别运行在2个虚拟机，consul通过check脚本检查2个容器，并将其IP和端口自动更新到consul-template中，虚拟机1输出一个文件。最后利用虚拟机1的Nginx反向代理进行域名访问2个容器。

对比单机上的测试，目的在于验证consul是否能发现非本机所运行的服务，并且通过template渲染IP端口到nginx配置文件。

此外，可以通过访问不同的域名，这些域名指向同一IP（虚拟机1），但反向代理到不同的IP和端口上。即只对外暴露虚拟机1的IP即可。

准备工作：修改docker用户权限，修改主机名以区分nomad结点

1 Nomad与Consul连接

1.1 Nomad

虚拟机1创建/home/.../nomad.d/nomad.hcl：

datacenter = "dc1"
data_dir = "/home/.../nomad/data"server {enabled = truebootstrap_expect = 2server_join {retry_join = ["192.168.10.11:4648"，"192.168.10.22:4648"]}
}client {enabled = trueservers = ["192.168.10.11:4647"]
}

虚拟机2创建/home/.../nomad.d/nomad.hcl：

datacenter = "dc1"
data_dir = "/home/.../nomad/data"server {enabled = truebootstrap_expect = 2server_join {retry_join = ["192.168.10.11:4648"，"192.168.10.22:4648"]}
}client {enabled = trueservers = ["192.168.10.22:4647"]
}

分别执行nomad agent -config /home/.../nomad.d启动nomad agent。

nomad server members和nomad node status可以查看状态。

1.2 Consul

虚拟机1创建/home/.../consul.d/consul.hcl：

{"datacenter": "dc1","data_dir": "/home/.../consul/data","log_level": "INFO","node_name": "consul_server1","server": true,"bind_addr": "192.168.10.11","client_addr": "0.0.0.0","bootstrap_expect":2,"ui": true,"enable_script_checks": true,
}

虚拟机2创建/home/.../consul.d/consul.hcl：

{"datacenter": "dc1","data_dir": "/home/.../consul/data","log_level": "INFO","node_name": "consul_server2","server": true,"bind_addr": "192.168.10.22","client_addr": "0.0.0.0","bootstrap_expect":2,"ui": true,"enable_script_checks": true,
}

虚拟机1创建/home/.../consul.d/http-echo.hcl：

service {name = "http-echo"port = 5678check {args = ["/home/.../consul/check.sh"]interval = "3s"}
}

虚拟机2创建/home/.../consul.d/http-echo2.hcl：

service {name = "http-echo2"port = 8765check {args = ["/home/.../consul/check2.sh"]interval = "3s"}
}

虚拟机1创建/home/.../consul/check.sh文件作为consul检测http-echo的脚本，注意chmod给权限：

#!/bin/bash
http=`docker ps | grep "http-echo" | grep "5678->5678"`if [ -n "$http" ];#非空thenexit 0elseexit 2fi

虚拟机2创建/home/.../consul/check2.sh文件作为consul检测http-echo2的脚本，注意chmod给权限：

#!/bin/bash
http=`docker ps | grep "http-echo" | grep "8765->8765"`if [ -n "$http" ];#非空thenexit 0elseexit 2fi

exit 0表示检测成功，1表示警告，其他表示失败。

分别执行consul agent -config-dir /home/.../consul.d启动consul agent。

consul members或浏览器访问192.168.10.11:8500查看状态。

1.3 Nomad Job(http-echo容器)

虚拟机1创建http-echo.nomad以备之后用：

job "http-echo" {datacenters = ["dc1"]constraint {attribute = "${node.unique.name}"value     = "ubuntu1"}group "example" {count = 1network {port "http" {static = "5678"}}task "server" {driver = "docker"config {image = "hashicorp/http-echo"ports = ["http"]args = ["-listen",":5678","-text","hello world",]}}}
}

虚拟机2创建http-echo2.nomad以备之后用：

job "http-echo2" {datacenters = ["dc1"]constraint {attribute = "${node.unique.name}"value     = "ubuntu2"}group "example2" {count = 1network {port "http2" {static = "8765"}}task "server2" {driver = "docker"config {image = "hashicorp/http-echo"ports = ["http2"]args = ["-listen",":8765","-text","hello world222",]}}}
}

constraint是限定参数，${node.unique.name}表示限定节点名唯一，即只能在特定结点运行job。目的是保证两个Job分别在两个虚拟机运行。

2 consul-template

源码：https://github.com/hashicorp/consul-template

二进制文件：https://releases.hashicorp.com/consul-template/

参考：configuration-file，Templating Language

虚拟机分别给权限：

sudo cp ./consul-template /usr/bin/consul-template
sudo chmod 777 /usr/bin/consul-template

虚拟机1创建两个template配置文件consul-template-config.hcl和consul-template-config2.hcl ：

consul {address = "192.168.10.11:8500"retry {enabled  = trueattempts = 12backoff  = "1s"}
}
template {source      = "/home/.../consul/http-echo.ctmpl"destination = "/home/.../consul/ip_port.txt"perms       = 0600command     = "/home/.../consul/template.sh"
}

consul {address = "192.168.10.22:8500"retry {enabled  = trueattempts = 12backoff  = "1s"}
}
template {source      = "/home/.../consul/http-echo2.ctmpl"destination = "/home/.../consul/ip_port2.txt"perms       = 0600command     = "/home/.../consul/template2.sh"
}

source表示模版文件，destination表示输出文件，perms表示文件访问权限，command表示每次变化执行的命令。

虚拟机1创建2个模版文件/home/.../consul/http-echo.ctmpl和/home/.../consul/http-echo2.ctmpl：

{{range service "http-echo"}}server  {{.Address}}:{{.Port}};{{end}}

{{range service "http-echo2"}}server  {{.Address}}:{{.Port}};{{end}}

创建空文件/home/.../consul/ip_port.txt、/home/.../consul/ip_port2.txt和/home/.../consul/ng.conf、/home/.../consul/ng2.conf，或者perms改成0644允许自动创建。

虚拟机1创建两个template脚本/home/.../consul/template.sh和/home/.../consul/template2.sh如下，注意chmod给权限：

#!/bin/bash
txt=`cat ./ip_port.txt`if [[  $txt =~ "server" ]];thenecho "upstream http_backend {" | cat > ./ng.confcat ./ip_port.txt >> ./ng.confecho "
}server {listen 80;server_name test.com;location / {proxy_pass http://http_backend;proxy_redirect off;proxy_set_header Host \$host;proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;proxy_set_header X-Real-IP \$remote_addr;}
}"  | cat >> ./ng.confecho "a" | sudo -S service nginx reloadelse> ./ng.conf
fi

#!/bin/bash
txt=`cat ./ip_port2.txt`if [[  $txt =~ "server" ]];thenecho "upstream http_backend2 {" | cat > ./ng2.confcat ./ip_port2.txt >> ./ng2.confecho "
}server {listen 80;server_name test2.com;location / {proxy_pass http://http_backend2;proxy_redirect off;proxy_set_header Host \$host;proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;proxy_set_header X-Real-IP \$remote_addr;}
}"  | cat >> ./ng2.confecho "a" | sudo -S service nginx reloadelse> ./ng2.conf
fi

因为如果upstream中没有server，service nginx reload会报错，所以需要判断一下输出文件是否有内容，没有的话就把nginx反向代理的配置文件设为空。

执行consul-template:

consul-template \-config=consul-template-config.hcl \-config=consul-template-config2.hcl

3 Nginx反向代理

安装Nginx：https://blog.csdn.net/weixin_43739110/article/details/121079232

在虚拟机1中/etc/nginx/nginx.conf加：

...
http{...include /home/.../consul/*.conf;...
}
...

service nginx restart重启Nginx。

nomad job run http-echo.nomad和nomad job run http-echo2.nomad启动之前创建的两个Nomad job容器。

浏览器访问192.168.10.11:8500可以结果。

运行容器后，会看到ip_port.txt和ip_port2.txt中的内容变为：

server  192.168.10.11:5678;

server  192.168.10.22:8765;

/home/.../consul/ng.conf中的内容变为：

upstream http_backend {server  192.168.10.11:5678;}server {listen 80;server_name test.com;location / {proxy_pass http://http_backend;proxy_redirect off;proxy_set_header Host $host;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Real-IP $remote_addr;}
}

/home/.../consul/ng2.conf中的内容变为：

upstream http_backend2 {server  192.168.10.22:8765;}server {listen 80;server_name test2.com;location / {proxy_pass http://http_backend2;proxy_redirect off;proxy_set_header Host $host;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Real-IP $remote_addr;}
}

/etc/hosts文件中添加：

192.168.10.11  test.com
192.168.10.22  test2.com

浏览器登陆test.com和test2.com可以看到（或执行curl test.com、curl test2.com命令）：

nomad job stop http-echo关闭Nomad job，会看到/home/.../consul/ip_port.txt和/home/.../consul/ng.conf中的内容消失。

浏览器登陆test.com会502错误。