永中文档在线转换预览双活实现方案

永中文档在线转换预览服务是永中软件股份有限公司基于十多年核心技术积累、面向移动互联领域推出的一款文档处理软件。永中采用自主可控核心技术，具备快速技术和服务响应能力，把文档原样输出为 HTML，图片等，即点即得、无需下载、保护文档隐私，快速高效，轻松实现文档在线安全阅读。

实现目标

通过nginx实现业务fcsserver负载均衡，后端可以动态扩展应用服务器

nginx通过keepalived实现高可用，解决单点故障

通过keepalived实现nginx双活配置，解决nginx主备资源使用率50%问题

通过dns轮询解析域名到双活vip，达到负载均衡效果

后期nginx达到瓶颈应考虑lvs+keepalived+nginx架构，动态扩展nginx服务器

环境准备

服务器可以连通外网或者有内网yum源服务器，本次实验服务器可连通外网

nginx01和nginx02服务器需要在同一网段的网络内

保证各服务器之前网络互通

保证服务器的防火墙和selinux关闭

必须对外提供域名访问，否则只能使用其中一个vip

内网部署DNS服务器，模拟DNS轮询解析

1、网络架构

2、实验服务器分布

主机	ip	操作系统	软件	端口	vip
nginx01	192.168.56.101	Centos7.6	nginx keepalived	80	192.168.56.200
nginx02	192.168.56.106	Centos7.6	nginx keepalived	80	192.168.56.201
fcs01	192.168.56.101	Centos7.6	tomcat	8080
fcs02	192.168.56.106	Centos7.6	tomcat	8080
共享存储	192.168.56.108	Centos7.6	nfs
缓存	192.168.56.108	Centos7.6	redis	6379
内部DNS	192.168.56.108	Centos7.6	bind	53

3、redis

部署服务器：

192.168.56.108

3.1 配置redis

/etc/redis.conf

# 监听地址
bind 0.0.0.0
# 认证密码
requirepass yozosoft

3.2 启动redis

systemctl enable redis --now && systemctl status redis

4、nfs

4.1 配置nfs

192.168.56.108（服务端）

# 创建存储目录
mkdir -p /opt/yozo/data
# 修改权限
chown -R nfsnobody.nfsnobody /opt/yozo/data
# 修改配置文件
vim /etc/exports/opt/yozo/data 192.168.56.0/24(rw,sync,all_squash)

192.168.56.101/192.168.56.106（客户端）

# 创建挂载点
mkdir -p /opt/yozo/data
# 挂载nfs共享目录
mount -t nfs 192.168.56.108:/opt/yozo/data /opt/yozo/data

4.2 启动nfs

systemctl enable rpcbind nfs --now && systemctl status rpcbind nfs

5、fcsserver

部署服务器：

192.168.56.101（fcsserver01）

192.168.56.106（fcsserver02）

5.1 部署fcs

以tomcat为中间件，本次实验fcs安装目录/opt/yozo/fcsserver/webapps/fcsserver，/opt/yozo/fcsserver为解压后的tomcat

部署项目包

mkdir opt/yozo/fcsserver/webapps/fcsserver -p
unzip fcscloud.war -d opt/yozo/fcsserver/webapps/fcsserver

修改fcsserver配置文件

# /opt/yozo/fcsserver/webapps/fcsserver/WEB-INF/classes/config.properties
inputDir=/opt/yozo/data/fcsdata/input  # 必须指定挂载共享存储目录的
outputDir=/opt/yozo/data/fcsdata/output
# 如果不提供域名，就只能配置其中一个vip；如果配置为域名，则缓存后，通过vip和fcsserver的ip将不能访问转换的缓存文件
viewDomain=http://www.fcsserver.com/fcsserver/

# /opt/yozo/fcsserver/webapps/fcsserver/WEB-INF/classes/application.yml
cache:                            #采用哪种缓存数据方式type: redis                     #local,redis,mysql(local模式只适用于单机,集群部署不支持)redis:enable: true                    #redis开关(预览设置权限时需要使用redis,并发和异步waitting机制使用redis,必开)redis:database: 1 # Redis数据库索引（默认为0）timeout: 10000 #设置客户端超时时间，单位是毫秒，默认为2000password: yozosoft #密码#单机版host: 192.168.56.108port: 6379

# /opt/yozo/fcsserver/webapps/fcsserver/WEB-INF/classes/
<property name="LOG_HOME">../logs</property>

5.2 配置fcs系统服务

配置fcsserver.service

# cat /usr/lib/systemd/system/fcsserver.service
[Unit]
Description=fcsserver Service.
After=network.target[Service]
Type=forking
Environment="PATH=/opt/yozo/jdk-8u251-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
WorkingDirectory=/opt/yozo/fcsserver/bin
ExecStart=/opt/yozo/fcsserver/bin/startup.sh
Restart=always
PrivateTmp=true
# 可以指定相关用户启动fcsserver
# User=yozo
# Group=yozo
LimitNOFILE=65536[Install]
WantedBy=multi-user.target

如果以普通用户启动，需要将tomcat目录属主属组设置成相应账户

# 此次实验以yozo用户启动
chown -R yozo. /opt/yozo/fcsserver

5.3 启动fcs

systemctl enable fcsserver --now && systemctl status fcsserver

6、nginx

部署服务器：

192.168.56.101（nginx01）

192.168.56.106（nginx02）

6.1 配置nginx

/etc/nginx/conf.d/fcsserver.conf配置

server {....location ~ /fcsserver {add_header Cache-Control private,no-store,max-age=0,no-cache,must-revalidate,post-check=0,pre-check=0;proxy_redirect off;proxy_set_header Host $http_host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded_Proto "http";proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_request_buffering off;proxy_read_timeout 7200;proxy_send_timeout 7200;proxy_pass http://fcsserver;}....
}upstream fcsserver {server 192.168.56.101:8080 fail_timeout=60s;server 192.168.56.106:8080 fail_timeout=60s;keepalive 256;
}

6.2 启动nginx

systemctl enable nginx --now && systemctl status nginx

7、keepalived

部署服务器：

192.168.56.101（nginx01）

192.168.56.106（nginx02）

7.1 配置keepalived

7.1.1 nginx01配置

/etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {router_id proxy1
}
vrrp_script chk_nginx {script "/etc/keepalived/check_nginx.sh"interval 2weight 20fall 3rise 2
}
vrrp_instance VI_1 {state MASTERinterface enp0s3virtual_router_id 51priority 100                           advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.56.200                     }track_script {chk_nginx}
}
vrrp_instance VI_2 {state BACKUPinterface enp0s3virtual_router_id 52priority 90                            advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.56.201                     }track_script {chk_nginx}
}

7.1.2 nginx02配置

# 备份keepalived.conf
mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.confbak

/etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {router_id proxy2
}
vrrp_script chk_nginx {script "/etc/keepalived/check_nginx.sh"interval 2weight 20fall 3rise 2
}
vrrp_instance VI_1 {state BACKUPinterface enp0s8virtual_router_id 51priority 90advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.56.200}track_script {chk_nginx}
}
vrrp_instance VI_2 {state MASTERinterface enp0s8virtual_router_id 52priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.56.201}track_script {chk_nginx}
}

双机/etc/keepalived/check_nginx.sh

#!/bin/bash
#时间变量，用于记录日志
d=`date --date today +%Y%m%d_%H:%M:%S`
#计算nginx进程数量
n=`ps -C nginx --no-heading|wc -l`
#如果进程为0，则启动nginx，并且再次检测nginx进程数量，
#如果还为0，说明nginx无法启动，此时需要关闭keepalived
if [ $n -eq "0" ]; then/etc/init.d/nginx startn2=`ps -C nginx --no-heading|wc -l`if [ $n2 -eq "0"  ]; thenecho "$d nginx down,keepalived will stop" >> /var/log/check_ng.logsystemctl stop keepalivedfi
fi

7.2 启动keepalived

systemctl enable keepalived --now && systemctl status keepalived

nginx01

nginx02

7.3 模拟故障

7.3.1 nginx01手动关闭nginx

nginx故障后，keepalived会自动启动nginx

7.3.2 nginx01手动关闭keepalived

nginx01的vip将会绑定到nginx02上面，nginx02将会出现2个vip，2个vip均可以访问fcsserver

8、DNS服务器部署

用于模拟公网DNS轮询解析，实际使用中是在域名提供商处配置指向映射外网ip

8.1 配置dns

/etc/named.conf

zone "fcsserver.com" IN {type master;file "fcsserver.com.zone";
};zone "56.168.192.in-addr.arpa" IN {type master;file "192.168.56.zone";
};

/var/named/fcsserver.com.zone


$TTL    86400
@               IN SOA  tom jerry (                     ; tom & jerry 这两个参数本应是主机名和邮件地址，这里随便填写，没有问题42              ; serial (d. adams)3H              ; refresh15M             ; retry1W              ; expiry1D )            ; minimumIN NS           ns.fcsserver.com.            ; notice : don't forget the dot in the endIN MX 10        mail.fcsserver.com.
www             IN A            192.168.56.200
www             IN A            192.168.56.201
ns              IN A            192.168.56.108
mail            IN A            192.168.56.108

/var/named/192.168.56.zone

$TTL    86400
@       IN      SOA     ns.fcsserver.com. root (1997022700 ; Serial28800      ; Refresh14400      ; Retry3600000    ; Expire86400 )    ; MinimumIN      NS      ns.fcsserver.com.
200     IN      PTR     www.fcsserver.com.
201     IN      PTR     www.fcsserver.com.
108     IN      PTR     mail.fcsserver.com.
108     IN      PTR     ns.fcsserver.com.

修改权限

chown named. /var/named/ -R

8.2 启动dns服务

systemctl enable named --now && systemctl status named

9、验证

目标：本次实验有2个vip 192.168.56.200 和 192.168.56.201，需要验证www.fcsserver.com分别解析到2个vip上，并确认每个vip后端服务正常可用

9.1 客户端dns配置

测试期间，禁用其他网卡，只留虚拟机网卡

添加DNS

9.2 DNS缓存清理

9.3 域名访问测试

测试之前需要确认此次DNS解析是否指向所需测试的VIP，如果不是请刷新DNS缓存

需要测试www.fcsserver.com --> 192.168.56.200和www.fcsserver.com --> 192.168.56.201

文件转换测试

转换文件访问测试

注：fcsserver的配置文件中viewDomain配置为域名，故转换文件预览连接只能通过www.fcsserver.com可以正常访问，通过vip、nginx ip、fcsserver ip均不可以访问