k8s集群管理（一）

cfssl 工具

cfssl-certinfo 验证证书信息

用法：

cfssl-certinfo -cert xxx.pem
cfssl-certinfo -domain www.baidu.com

将证书还原成 json 结构

md5sum命令：
验证一个文件的md5值

md5sum 文件

从 .kubeconfig 配置文中反解获取证书

echo “LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR3RENDQXFpZ0F3SUJBZ0lVYkh1czBSZkE2dHA1TjRYZnYvRWhkSlA4dytvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1hqRUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjJKbGFXcHBibWN4RURBT0JnTlZCQWNUQjJKbAphV3BwYm1jeEN6QUpCZ05WQkFvVEFtOWtNUXd3Q2dZRFZRUUxFd052Y0hNeEVEQU9CZ05WQkFNVEIwWmxibWRaCmRXNHdIaGNOTWpFd09ESXpNRGt5TnpBd1doY05OREV3T0RFNE1Ea3lOekF3V2pCZk1Rc3dDUVlEVlFRR0V3SkQKVGpFUU1BNEdBMVVFQ0JNSFltVnBhbWx1WnpFUU1BNEdBMVVFQnhNSFltVnBhbWx1WnpFTE1Ba0dBMVVFQ2hNQwpiMlF4RERBS0JnTlZCQXNUQTI5d2N6RVJNQThHQTFVRUF4TUlhemh6TFc1dlpHVXdnZ0VpTUEwR0NTcUdTSWIzCkRRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRGNnS2M1VzNER3hjUklkeGR0OXh3aEdrWDdOVXREM21pamNqTmMKTnVYb21KYTlpMGlySU9ObVNyNm1GV05NZU14MXdmeW9aY2J4OVdSRXZHcTYwRW0xZmo3SWpyRDByVDZRa1Ivbwp2QkRpWFJ5dWdjMGZPYUVMcG01OWc3SDJXUkpFVjhOV1RXSnBpUTdNcUloYVJqOW1QODEzejlHZnJiZ0hzUnBlCkQ4RGRXem1ZNklXbjlRaHYxcnZ3U1ZDVVJsZ0tUS1pvd0VCclEwZk9BZ0U3Nm5ibDZXZktIRjB0SUpFcTY1ZU0KZngvbExJRExuRGpIWE5SMWorenBvWGZjNWlBNU9jQ1A0bGhUUlBMZW9CUUF1WWpjc3Z4SGY4UVMrSzRvTTVwNwpRWGRVc0RPUk16dW96TktCVFRKUElEZlpXOFZaNTA0eGo4L2UxOXNrS1pYRUFRYzFBZ01CQUFHamRUQnpNQTRHCkExVWREd0VCL3dRRUF3SUZvREFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUEKTUIwR0ExVWREZ1FXQkJSTjJDQnM0RWl5cjJOZDAzaFBGVEwwbFdPRXNUQWZCZ05WSFNNRUdEQVdnQlRnRnhmLwo4c0ozcmYxcm0xdUptN21nZGZFVmh6QU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFBajl5elI4c1JaODQ5VHVQCmV4VDdoNzFkRGxJc0N5aFQzL0pwRHJ3R0Z6L0s1ZFFkbXcyRVh1RWVacXpPYzlRSzZQMEJ1UG9JRTZTcCtvZXMKbnlvTWFUcE96Q0NjT1k1WlRZRzNtQUdiVlB6SlpMWFVIMnNiRDJpYkI4RzYzUlpVV080TG9jS2RqRm9NMjlyNQo4UGdlMEJGU2s4ZXFzam1GMFNITHNLeTFKZTRwU1ZDLzlzVGlpRHllTitwQk02NStSeHJxSWRpSSs2NVNma29vClpBSE5NYnJLNGo1Rlo3ZCtESk0xREZTcjA1V3lQY1lSVTB5bkg2UGd2My9zRHRHWSt0THRVckRiZnlySFZiZTEKczRzc3Bpd0lGQnc4Tk40bitNQmxkd1VxTFB3MGI5anJUbVlRcGQ3K0JIWkMvOFFVcGk2b0NLT1RJQWZkOUxqUApVa3NuV2c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==”|base64 -d > 123.pem

执行命令：
```
cfssl-certinfo -cert 123.pem
```

kubeconfig 文件

这是一个k8s用户的配置文件
它里面含有证书信息
证书过期或更换，需要同步替换该文件

kubectl

管理 K8S 核心资源的三种基本方法：

1. 陈述式管理方法 - 主要依赖命令行CLI工具进行管理

查看名称空间
```
kubectl get namespaces
kubectl get ns
```
查询 default 名称空间里所有的资源
```
kubectl get all -n default
```

创建/删除名称空间

kubectl create ns app
kubectl create namespace app
kubectl delete ns app
kubectl delete namespace app

创建 deployment 资源

kubectl create deployment nginx-dp --image=harbor.od.com/public/nginx:v1.7.9 -n kube-public # 在kube-public名称空间下创建一个pod 控制器，而控制器的类型是 deployment，根据镜像，将container 运行在pod中
kubectl get pods -n kube-public # 查看 kube-public名称空间下的 pods
kubectl get pods -n kube-public -o wide # 详细查看

查看 deployment 资源

kubectl get deployment -n kube-public  #如果不指定名称空间，使用 default 名称空间

详细查看 deployment

kubectl describe deployment nginx-dp -n kube-public

进入 pod

kubectl exec -it nginx-dp-5dfc689474-tflm4 /bin/bash -n kube-public # nginx-dp-5dfc689474-tflm4 pod的名字

删除 deployment

kubectl delete deploy nginx-dp -n kube-public

创建 service

kubectl expose deployment nginx-dp --port=80 -n kube-public
kubectl scale deployment nginx-dp --replicas=2 -n kube-public # 扩容deployment

查看 service

kubectl describe service nginx-dp -n kube-public

kubectl 是官方的CLI命令行工具，用于与 apiserver 进行通信，将用户在命令行输入的命令，组织并转化为apiserver能识别的信息，进而实现管理K8S各种资源的一种有效途径。
K8S集群管理资源的唯一入口是通过相应的方法调用apiserver的接口。
kubectl 对于增加资源，删除资源，查看资源都很方便，但是对于资源修改，比较麻烦。

2. 声明式管理方法 - 主要依赖统一资源配置清单（manifest）进行管理

查看资源配置清单

kubectl get svc nginx-dp -o yaml -n kube-public

解释资源配置清单
```
kubectl explain service
```
应用资源配置清单
```
kubectl create/apply -f xxxx.yaml
```
在线修改资源配置清单
```
kubectl edit svc service_name
```

离线修改资源配置清单，并引用

vi xxx.yaml #修改资源配置清单
kubectl apply -f xxx.yaml # 修改后并应用资源配置清单

声明式删除一个资源
```
kubectl delete -f xxxx.yaml
```

3. GUI管理方法 - 主要依赖图形化操作界面进行管理

Kubernetes 网络模型

Kubernetes 设计了网络模型，但却将它的实现交给了网络插件，CNI网络插件最主要的功能就是实现POD资源能够跨宿主机进行通信。常见的CNI网络插件有：Flannel、Calico、Canal、Contiv、OpenContrail、NSX-T、Kube-router

部署K8S的CNI网络插件 - Flannel

集群规划

主机名	角色	ip
hdss7-21.host.com	flannel	10.4.7.21
hdss7-22.host.com	flannel	10.4.7.22

下载 flannel、解压、做软连接

解压，做软链接

mkdir -p /opt/flannel-v0.11.0
tar xf flannel-v0.11.0-linux-amd64.tar.gz -C /opt/flannel-v0.11.0/
ln -s /opt/flannel-v0.11.0/ /opt/flannel

flannel要连接etcd，需要将hdss7-200上的证书拷贝过来

mkdir -p /opt/flannel-v0.11.0/cert
cd /opt/flannel-v0.11.0/cert
scp hdss7-200:/opt/certs/ca.pem .
scp hdss7-200:/opt/certs/client.pem .
scp hdss7-200:/opt/certs/client-key.pem .

创建配置 /opt/flannel-v0.11.0/subnet.env

FLANNEL_NETWORK=172.7.0.0/16        # pod的网段
FLANNEL_SUBNET=172.7.21.1/24       # 本机运行pod的网段
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false

创建启动脚本 /opt/flannel-v0.11.0/flanneld.sh

创建启动脚本

#!/bin/sh
./flanneld \--public-ip=10.4.7.21 \--etcd-endpoints=https://10.4.7.12:2379,https://10.4.7.21:2379,https://10.4.7.22:2379 \--etcd-keyfile=./cert/client-key.pem \--etcd-certfile=./cert/client.pem \--etcd-cafile=./cert/ca.pem \--iface=ens33 \--subnet-file=./subnet.env \--healthz-port=2401

增加执行权限，并创建日志目录

chmod +x  /opt/flannel-v0.11.0/flanneld.sh
mkdir -p /data/logs/flanneld

操作etcd，增加host-gw

cd /opt/etcd/
./etcdctl set /coreos.com/network/config '{"Network": "172.7.0.0/16", "Backend": {"Type": "host-gw"}}'
./etcdctl member list # 查看etcd节点
./etcdctl get /coreos.com/network/config # 查看网络

创建 supervisor 配置

创建 /etc/supervisord.d/flannel.ini

[program:flanneld-7-21]
command=/opt/flannel/flanneld.sh                             ; the program (relative uses PATH, can take args)
numprocs=1                                                   ; number of processes copies to start (def 1)
directory=/opt/flannel                                       ; directory to cwd to before exec (def no cwd)
autostart=true                                               ; start at supervisord start (default: true)
autorestart=true                                             ; retstart at unexpected quit (default: true)
startsecs=30                                                 ; number of secs prog must stay running (def. 1)
startretries=3                                               ; max # of serial start failures (default 3)
exitcodes=0,2                                                ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                              ; signal used to kill process (default TERM)
stopwaitsecs=10                                              ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                    ; setuid to this UNIX account to run the program
redirect_stderr=true                                         ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/flanneld/flanneld.stdout.log       ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                 ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                     ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                  ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                  ; emit events on stdout writes (default false)

启动flannel 插件

supervisorctl update
supervisorctl start flannel-7-21

Flannel 的 host-gw 模型

Flannel 的 VxLan 模型

iptables，解决集群内部的 SNAT 转换问题

安装iptables
```
yum install iptables-services -y
```

启动iptables并设置开机自启动

systemctl start iptables
systemctl enable iptables

优化 iptables 规则

iptables-save |grep -i postrouting # 查看iptables规则
iptables -t nat -D POSTROUTING -s 172.7.21.0/24 ! -o docker0 -j MASQUERADE # 删除一条iptables规则 -D表示删除
# 插入一条规则 -I 表示插入 -s 表示源地址 ! 表示否定：插入一条，源地址网络为172.7.21.0/24网络地址，不是去往172.7.0.0/16
# 这个网络，也不是从docker0出网，那么才做网络地址转换
iptables -t nat -I POSTROUTING -s 172.7.21.0/24 ! -d 172.7.0.0/16 ! -o docker0 -j MASQUERADE
iptables-save > /etc/sysconfig/iptables #将规则保存

10.4.7.21主机上，来源是172.7.21.0/24段的docker的ip，目标ip不是172.7.0.0/16段，网络发包不从docker0网桥设备出站的，才进行SNAT转换。

iptables-save |grep -i reject # 查看iptables的拒绝规则
iptables -t filter -D INPUT -j REJECT --reject-with icmp-host-prohibited # 删除 INPUT 链拒绝规则
iptables -t filter -D FORWARD -j REJECT --reject-with icmp-host-prohibited # 删除 FORWARD 链拒绝规则
iptables-save > /etc/sysconfig/iptables

服务发现

简单来说，服务发现就是服务（应用）之间相互定位的过程。
在K8S集群里，POD的IP是不断变换的，如何”以不变应万变“？
1. 抽象出Service资源，通过标签选择器，关联一组POD。
2. 抽象出集群网络，通过相对固定的”集群IP“，使服务接入点固定。

部署K8S的服务发现插件 - CoreDNS

部署coredns (在运维主机上hdss7-200上执行）

docker pull coredns/coredns:1.6.1 # 从docker仓库中拉取镜像
docker tag coredns/coredns:1.6.1 harbor.od.com/public/coredns:v1.6.1 # 给镜像打tag
docker push harbor.od.com/public/coredns:v1.6.1 # 将镜像推送到私有仓库

运维主机上配置nginx（hdss7-200上执行）

mkdir /data/k8s-yaml
vi /etc/nginx/conf.d/k8s-yaml.od.com.conf
server {listen       80;server_name  k8s-yaml.od.com;location / {autoindex on;default_type text/plain;root /data/k8s-yaml;}
}
# 保存退出后，执行下面命令
nginx -t
nginx -s reload

配置内网DNS解析（hdss7-11上执行）

vi /var/named/od.com.zone$ORIGIN od.com.
$TTL 600        ; 10 minutes
@       IN SOA  dns.od.com. dnsadmin.od.com. (2021082205 ; serial  # 注意这里的序列化增加1，每次编辑改文件，序列化增加110800      ; refresh (3 hours)900        ; retry (15 minutes)604800     ; expire (1 week)86400      ; minimum (1 day))NS    dns.od.com.
$TTL 60 ; 1 minute
dns                 A     10.4.7.11
harbor              A     10.4.7.200
k8s-yaml            A     10.4.7.200  # 添加一条 DNS解析规则

在运维主机上 /data/k8s-yaml目录下，编辑文件rbac.yaml、ConfigMap.yaml、Deployment.yaml、Service.yaml资源文件

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:name: corednsnamespace: kube-systemlabels:kubernetes.io/cluster-service: "true"addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:kubernetes.io/bootstrapping: rbac-defaultsaddonmanager.kubernetes.io/mode: Reconcilename: system:coredns
rules:
- apiGroups:- ""resources:- endpoints- services- pods- namespacesverbs:- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsaddonmanager.kubernetes.io/mode: EnsureExistsname: system:coredns
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:coredns
subjects:
- kind: ServiceAccountname: corednsnamespace: kube-system

cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:name: corednsnamespace: kube-system
data:Corefile: |.:53 {errorsloghealthreadykubernetes cluster.local 10.254.0.0/16        # service网段forward . 10.4.7.11                      # 物理机安装dns服务的地址     cache 30loopreloadloadbalance}

dp.yaml

apiVersion: apps/v1
kind: Deployment
metadata:name: corednsnamespace: kube-systemlabels:k8s-app: corednskubernetes.io/name: "CoreDNS"
spec:replicas: 1selector:matchLabels:k8s-app: corednstemplate:metadata:labels:k8s-app: corednsspec:priorityClassName: system-cluster-criticalserviceAccountName: corednscontainers:- name: corednsimage: harbor.od.com/public/coredns:v1.6.1args:- -conf- /etc/coredns/CorefilevolumeMounts:- name: config-volumemountPath: /etc/corednsports:- containerPort: 53name: dnsprotocol: UDP- containerPort: 53name: dns-tcpprotocol: TCP- containerPort: 9153name: metricsprotocol: TCPlivenessProbe:httpGet:path: /healthport: 8080scheme: HTTPinitialDelaySeconds: 60timeoutSeconds: 5successThreshold: 1failureThreshold: 5dnsPolicy: Defaultvolumes:- name: config-volumeconfigMap:name: corednsitems:- key: Corefilepath: Corefile

svc.yaml

apiVersion: v1
kind: Service
metadata:name: corednsnamespace: kube-systemlabels:k8s-app: corednskubernetes.io/cluster-service: "true"kubernetes.io/name: "CoreDNS"
spec:selector:k8s-app: corednsclusterIP: 192.168.0.2        # dns服务的ipports:- name: dnsport: 53protocol: UDP- name: dns-tcpport: 53- name: metricsport: 9153protocol: TCP

在hdss7-21上执行，创建

kubectl apply -f http://k8s-yaml.od.com/coredns/rbac.yaml
kubectl apply -f http://k8s-yaml.od.com/coredns/cm.yaml
kubectl apply -f http://k8s-yaml.od.com/coredns/dp.yaml
kubectl apply -f http://k8s-yaml.od.com/coredns/svc.yaml

K8S服务暴露的两种方式

K8S的DNS实现了服务在集群“内”被自动发现，那么如何使得服务在K8S集群“外”被使用和访问呢？

使用 NodePort 型的 Service，该方式无法使用 Kube-proxy 的 ipvs 模型，只能使用 iptables 模型。
使用 Ingress 资源。Ingress 只能调度并暴露7层应用，特指http和https协议。

Ingress 是 K8S API的标准资源类型之一，也是一种核心资源。它其实就是一组基于域名和URL路径，把用户的请求转发至指定Service 资源的规则。
Ingress 可以将集群外部的请求流量，转发至集群内部，从而实现“服务暴露”。
Ingress 控制器是能够为 Ingress 资源监听某套接字，然后根据 Ingress 规则匹配机制路由调度流量的一个组件。

部署traefik （ingress 控制器）

在运维主机hdss7-200 上，准备traefik镜像

docker pull traefik:v1.7.2-alpine
docker tag traefik:v1.7.2-alpine harbor.od.com/public/traefik:v1.7.2
docker push harbor.od.com/public/traefik:v1.7.2

创建资源配置清单

cd /data/k8s-yaml/
mkdir traefik
cd traefik/
vim rbac.yaml

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:name: traefik-ingress-controllernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:name: traefik-ingress-controller
rules:- apiGroups:- ""resources:- services- endpoints- secretsverbs:- get- list- watch- apiGroups:- extensionsresources:- ingressesverbs:- get- list- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: traefik-ingress-controller
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: traefik-ingress-controller
subjects:
- kind: ServiceAccountname: traefik-ingress-controllernamespace: kube-system

ds.yaml文档

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:name: traefik-ingressnamespace: kube-systemlabels:k8s-app: traefik-ingress
spec:template:metadata:labels:k8s-app: traefik-ingressname: traefik-ingressspec:serviceAccountName: traefik-ingress-controllerterminationGracePeriodSeconds: 60containers:- image: harbor.od.com/public/traefik:v1.7.2name: traefik-ingressports:- name: controllercontainerPort: 80hostPort: 81- name: admin-webcontainerPort: 8080securityContext:capabilities:drop:- ALLadd:- NET_BIND_SERVICEargs:- --api- --kubernetes- --logLevel=INFO- --insecureskipverify=true- --kubernetes.endpoint=https://10.4.7.10:7443        # keepalive VIP的地址- --accesslog- --accesslog.filepath=/var/log/traefik_access.log- --traefiklog- --traefiklog.filepath=/var/log/traefik.log- --metrics.prometheus

svc.yaml

kind: Service
apiVersion: v1
metadata:name: traefik-ingress-servicenamespace: kube-system
spec:selector:k8s-app: traefik-ingressports:- protocol: TCPport: 80name: controller- protocol: TCPport: 8080name: admin-web

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:name: traefik-web-uinamespace: kube-systemannotations:kubernetes.io/ingress.class: traefik
spec:rules:- host: traefik.od.comhttp:paths:- path: /backend:serviceName: traefik-ingress-serviceservicePort: 8080

应用资源配置清单

kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml
kubectl apply -f http://k8s-yaml.od.com/traefik/ds.yaml
kubectl apply -f http://k8s-yaml.od.com/traefik/svc.yaml
kubectl apply -f http://k8s-yaml.od.com/traefik/ingress.yaml

在HDSS7-11 和 HDSS7-12两台主机上（配置了keepalive的主机上）的nginx配置反向代理

vi /etc/nginx/conf.d/od.com.confupstream default_backend_traefik {server 10.4.7.21:81    max_fails=3 fail_timeout=10s; # 此ip为node ip+81端口，每个node节点都需要加上server 10.4.7.22:81    max_fails=3 fail_timeout=10s;
}
server {server_name *.od.com;                   # 泛域名匹配，凡是od.con的域名内的http服务。都给到ingress里面location / {proxy_pass http://default_backend_traefik;proxy_set_header Host       $http_host;proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;}
}