[kubernetes]-kubernetes+nfs创建高可用mysql

安装mysql主从玩一下。

1 创建rabc rabc-1.yaml

---
# 唯一需要修改的地方只有namespace,根据实际情况定义
apiVersion: v1
kind: ServiceAccount
metadata:name: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default        #根据实际环境设定namespace,下面类同
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: nfs-client-provisioner-runner
rules:- apiGroups: [""]resources: ["persistentvolumes"]verbs: ["get", "list", "watch", "create", "delete"]- apiGroups: [""]resources: ["persistentvolumeclaims"]verbs: ["get", "list", "watch", "update"]- apiGroups: ["storage.k8s.io"]resources: ["storageclasses"]verbs: ["get", "list", "watch"]- apiGroups: [""]resources: ["events"]verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: run-nfs-client-provisioner
subjects:- kind: ServiceAccountname: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default
roleRef:kind: ClusterRolename: nfs-client-provisioner-runnerapiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default
rules:- apiGroups: [""]resources: ["endpoints"]verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisioner
subjects:- kind: ServiceAccountname: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default
roleRef:kind: Rolename: leader-locking-nfs-client-provisionerapiGroup: rbac.authorization.k8s.io

2 创建nfsStorageClass nfs-StorageClass-2.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: managed-nfs-storage
provisioner: mysql-nfs-storage #这里的名称要和provisioner配置文件中的环境变量PROVISIONER_NAME保持一致parameters:  archiveOnDelete: "false"

这里需要先创建好nfs server

3 创建nfs-provisioner nfs-provisioner-3.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: nfs-client-provisionerlabels:app: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default  #与RBAC文件中的namespace保持一致
spec:replicas: 1selector:matchLabels:app: nfs-client-provisionerstrategy:type: Recreateselector:matchLabels:app: nfs-client-provisionertemplate:metadata:labels:app: nfs-client-provisionerspec:serviceAccountName: nfs-client-provisionercontainers:- name: nfs-client-provisionerimage: quay.io/external_storage/nfs-client-provisioner:latestvolumeMounts:- name: nfs-client-rootmountPath: /persistentvolumesenv:- name: PROVISIONER_NAMEvalue: mysql-nfs-storage  #provisioner名称,请确保该名称与 nfs-StorageClass.yaml文件中的provisioner名称保持一致- name: NFS_SERVERvalue: 192.168.1.171   #NFS Server IP地址- name: NFS_PATHvalue: /srv/data    #NFS挂载卷volumes:- name: nfs-client-rootnfs:server: 192.168.1.171  #NFS Server IP地址path: /srv/data     #NFS 挂载卷

4 测试一下创建test-claim-4.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:name: test-claimannotations:volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"   #与nfs-StorageClass.yaml metadata.name保持一致
spec:accessModes:- ReadWriteManyresources:requests:storage: 1Mi

5 测试一下创建test-pod-5.yaml

kind: Pod
apiVersion: v1
metadata:name: test-pod
spec:containers:- name: test-podimage: busybox:1.24command:- "/bin/sh"args:- "-c"- "touch /mnt/SUCCESS && exit 0 || exit 1"   #创建一个SUCCESS文件后退出volumeMounts:- name: nfs-pvcmountPath: "/mnt"restartPolicy: "Never"volumes:- name: nfs-pvcpersistentVolumeClaim:claimName: test-claim  #与PVC名称保持一致

在nfs 生成了SUCCESS文件

6 测试一下 nginx-statefulset-6.yaml

# 这个我自己没测试
---
apiVersion: v1
kind: Service
metadata:name: nginx-headlesslabels:app: nginx
spec:ports:- port: 80name: webclusterIP: None   #注意此处的值,None表示无头服务selector:app: nginx
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:name: web
spec:serviceName: "nginx"replicas: 2  #两个副本template:metadata:labels:app: nginxspec:containers:- name: nginximage: ikubernetes/myapp:v1ports:- containerPort: 80name: webvolumeMounts:- name: wwwmountPath: /usr/share/nginx/htmlvolumeClaimTemplates:- metadata:name: wwwannotations:volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"   #managed-nfs-storage为我们创建的storage-class名称spec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 1Gi

7 创建mysql配置文件 cm-mysql-7.yaml

apiVersion: v1
kind: ConfigMap
metadata:name: mysqllabels:app: mysql
data:master.cnf: |# Apply this config only on the master.[mysqld]log-binlog_bin_trust_function_creators=1lower_case_table_names=1slave.cnf: |# Apply this config only on slaves.[mysqld]super-read-onlylog_bin_trust_function_creators=1

8 创建mysql 的svc svc-mysql-8.yaml


# Headless service for stable DNS entries of StatefulSet members.
apiVersion: v1
kind: Service
metadata:name: mysqllabels:app: mysql
spec:ports:- name: mysqlport: 3306clusterIP: Noneselector:app: mysql
---
# Client service for connecting to any MySQL instance for reads.
# For writes, you must instead connect to the master: mysql-0.mysql.
apiVersion: v1
kind: Service
metadata:name: mysql-readlabels:app: mysql
spec:ports:- name: mysqlport: 3306selector:app: mysql

可以看到，这两个 Service 都代理了所有携带 app=mysql 标签的 Pod，也就是所有的 MySQL Pod。端口映射都是用 Service 的 3306 端口对应 Pod 的 3306 端口。

不同的是，第一个名叫“mysql”的 Service 是一个 Headless Service（即：clusterIP= None）。所以它的作用，是通过为 Pod 分配 DNS 记录来固定它的拓扑状态，比如“mysql-0.mysql”和“mysql-1.mysql”这样的 DNS 名字。其中，编号为 0 的节点就是我们的主节点。

而第二个名叫“mysql-read”的 Service，则是一个常规的 Service。

并且我们规定，所有用户的读请求，都必须访问第二个 Service 被自动分配的 DNS 记录，即：“mysql-read”（当然，也可以访问这个 Service 的 VIP）。这样，读请求就可以被转发到任意一个 MySQL 的主节点或者从节点上。

备注：Kubernetes 中的所有 Service、Pod 对象，都会被自动分配同名的 DNS 记录。并且读请求应该是分配到所有mysql节点，不是网上博客说的读是到read的slave是那个。

而所有用户的写请求，则必须直接以 DNS 记录的方式访问到 MySQL 的主节点，也就是：“mysql-0.mysql“这条 DNS 记录。

9 创建mysql的sts sts-mysql-9.yaml


apiVersion: apps/v1
kind: StatefulSet
metadata:name: mysql
spec:selector:matchLabels:app: mysqlserviceName: mysqlreplicas: 3volumeClaimTemplates:- metadata:name: dataannotations:volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"spec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 10Gitemplate:metadata:labels:app: mysqlspec:initContainers:- name: init-mysqlimage: registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7command:- bash- "-c"- |set -ex[[ `hostname` =~ -([0-9]+)$ ]] || exit 1ordinal=${BASH_REMATCH[1]}echo [mysqld] > /mnt/conf.d/server-id.cnfecho server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnfif [[ $ordinal -eq 0 ]]; thencp /mnt/config-map/master.cnf /mnt/conf.d/elsecp /mnt/config-map/slave.cnf /mnt/conf.d/fivolumeMounts:- name: confmountPath: /mnt/conf.d- name: config-mapmountPath: /mnt/config-map- name: clone-mysqlimage: registry.cn-hangzhou.aliyuncs.com/xjm_public/xtrabackup:1.0command:- bash- "-c"- |set -ex[[ -d /var/lib/mysql/mysql ]] && exit 0[[ `hostname` =~ -([0-9]+)$ ]] || exit 1ordinal=${BASH_REMATCH[1]}[[ $ordinal -eq 0 ]] && exit 0ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysqlxtrabackup --prepare --target-dir=/var/lib/mysqlvolumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dcontainers:- name: mysqlimage: registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7env:- name: MYSQL_ALLOW_EMPTY_PASSWORDvalue: "1"ports:- name: mysqlcontainerPort: 3306volumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dresources:requests:cpu: 500mmemory: 1GilivenessProbe:exec:command: ["mysqladmin", "ping"]initialDelaySeconds: 30periodSeconds: 10timeoutSeconds: 5readinessProbe:exec:command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]initialDelaySeconds: 5periodSeconds: 2timeoutSeconds: 1- name: xtrabackupimage: registry.cn-hangzhou.aliyuncs.com/xjm_public/xtrabackup:1.0ports:- name: xtrabackupcontainerPort: 3307command:- bash- "-c"- |set -excd /var/lib/mysqlif [[ -f xtrabackup_slave_info ]]; thenmv xtrabackup_slave_info change_master_to.sql.inrm -f xtrabackup_binlog_infoelif [[ -f xtrabackup_binlog_info ]]; then[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1rm xtrabackup_binlog_infoecho "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.infiif [[ -f change_master_to.sql.in ]]; thenecho "Waiting for mysqld to be ready (accepting connections)"until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; doneecho "Initializing replication from clone position"mv change_master_to.sql.in change_master_to.sql.origmysql -h 127.0.0.1 <<EOF$(<change_master_to.sql.orig),MASTER_HOST='mysql-0.mysql',MASTER_USER='root',MASTER_PASSWORD='',MASTER_CONNECT_RETRY=10;START SLAVE;EOFfiexec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"volumeMounts:- name: datamountPath: /var/lib/mysqlsubPath: mysql- name: confmountPath: /etc/mysql/conf.dresources:requests:cpu: 100mmemory: 100Mivolumes:- name: confemptyDir: {}- name: config-mapconfigMap:name: mysql

测试服务启动可能比较慢

kubectl run mysql-client --image=registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7 -it --rm --restart=Never -- mysql -h mysql-0.mysql.default

CREATE DATABASE demo;
CREATE TABLE demo.messages (message VARCHAR(250));
INSERT INTO demo.messages VALUES ('hello');

查看主机名为mysql-read的数据

kubectl run mysql-client2 --image=registry.cn-hangzhou.aliyuncs.com/xjm_public/mysql:5.7 -i -t --rm --restart=Never -- mysql -h mysql-read.default

select * from demo.messages;

最后参考博客里的有些错误描述需要纠正一下

1 读请求应该是0-2这3台机子都会请求。因为svc设置的是app:mysql 主从并没有区分

2 删除master节点似乎并不会由slave 升级为master 依旧是重新生成了一个新的mysql-0

测试继续写入数据没有问题后续进行更多测试

但是slave不会升级成master的话可能可用性还有待提高更换为operater 主从用不同的sts来部署可能会好些

参考

mysql

https://my.oschina.net/u/3627276/blog/4528810

nfs storage

https://www.cnblogs.com/panwenbin-logs/p/12196286.html