转载地址：https://blog.csdn.net/qq_37950254/article/details/95204011

        <link rel="stylesheet" href="https://csdnimg.cn/release/phoenix/template/css/ck_htmledit_views-833878f763.css"><link rel="stylesheet" href="https://csdnimg.cn/release/phoenix/template/css/ck_htmledit_views-833878f763.css"><div class="htmledit_views" id="content_views"><h3><a name="t0"></a><a name="t0"></a>1、安装环境准备：</h3>

部署节点说明

IP地址 主机名 CPU 内存 磁盘
192.168.1.108 qas-k8s-master01 2 2G 30G
192.168.1.165 qas-k8s-node01 2 2G 30G
192.168.1.215 qas-k8s-node02 2 2G 30G

2.下载1.15的包

地址：https://kubernetes.io/docs/setup/release/notes/

或者下载百度网盘的：链接 提取码：op7d

2.1 下载tar.gz包到服务器并解压到master节点的root目录下



  1. 

    
            [root@qas-k8s-master01 bin]# pwd
        
    2. 

  2. 

    
            /root/kubernetes/server/bin
        
    3. 

  3. 

    
            [root@qas-k8s-master01 bin]# ll
        
    4. 

  4. 

    
            total 1570456
        
    5. 

  5. 

    
            -rwxr-xr-x 1 root root 42793600 Apr 9 01:51 apiextensions-apiserver
        
    6. 

  6. 

    
            -rwxr-xr-x 1 root root 100402880 Apr 9 01:51 cloud-controller-manager
        
    7. 

  7. 

    
            -rw-r--r-- 1 root root 8 Apr 9 01:47 cloud-controller-manager.docker_tag
        
    8. 

  8. 

    
            -rw-r--r-- 1 root root 144294400 Apr 9 01:47 cloud-controller-manager.tar
        
    9. 

  9. 

    
            -rwxr-xr-x 1 root root 211183808 Apr 9 01:51 hyperkube
        
    10. 

  10. 

    
            -rwxr-xr-x 1 root root 39587104 Apr 9 01:51 kubeadm
        
    11. 

  11. 

    
            -rwxr-xr-x 1 root root 167554400 Apr 9 01:51 kube-apiserver
        
    12. 

  12. 

    
            -rw-r--r-- 1 root root 8 Apr 9 01:47 kube-apiserver.docker_tag
        
    13. 

  13. 

    
            -rw-r--r-- 1 root root 211445760 Apr 9 01:47 kube-apiserver.tar
        
    14. 

  14. 

    
            -rwxr-xr-x 1 root root 115579424 Apr 9 01:51 kube-controller-manager
        
    15. 

  15. 

    
            -rw-r--r-- 1 root root 8 Apr 9 01:47 kube-controller-manager.docker_tag
        
    16. 

  16. 

    
            -rw-r--r-- 1 root root 159471104 Apr 9 01:47 kube-controller-manager.tar
        
    17. 

  17. 

    
            -rwxr-xr-x 1 root root 43115328 Apr 9 01:51 kubectl
        
    18. 

  18. 

    
            -rwxr-xr-x 1 root root 127940544 Apr 9 01:51 kubelet
        
    19. 

  19. 

    
            -rwxr-xr-x 1 root root 36685440 Apr 9 01:51 kube-proxy
        
    20. 

  20. 

    
            -rw-r--r-- 1 root root 8 Apr 9 01:47 kube-proxy.docker_tag
        
    21. 

  21. 

    
            -rw-r--r-- 1 root root 83982848 Apr 9 01:47 kube-proxy.tar
        
    22. 

  22. 

    
            -rwxr-xr-x 1 root root 39258304 Apr 9 01:51 kube-scheduler
        
    23. 

  23. 

    
            -rw-r--r-- 1 root root 8 Apr 9 01:47 kube-scheduler.docker_tag
        
    24. 

  24. 

    
            -rw-r--r-- 1 root root 83149824 Apr 9 01:47 kube-scheduler.tar
        
    25. 

  25. 

    
            -rwxr-xr-x 1 root root 1648224 Apr 9 01:51 mounter
        
    26. 

  26. 

    
            [root@qas-k8s-master01 bin]#
        




二、Kubernetes 安装及配置


1、初始化环境


1.1、设置关闭防火墙及SELINUX




  1. 

    
            systemctl stop firewalld && systemctl disable firewalld
        
    2. 

  2. 

    
            setenforce 0
        
    3. 

  3. 

    
            vi /etc/selinux/config
        
    4. 

  4. 

    
            SELINUX=disabled
        




1.2、关闭Swap




  1. 

    
            swapoff -a && sysctl -w vm.swappiness=0
        
    2. 

  2. 

    
            vi /etc/fstab
        
    3. 

  3. 

    
            #UUID=7bff6243-324c-4587-b550-55dc34018ebf swap swap defaults 0 0
        




1.3、设置Docker所需参数




  1. 

    
            cat << EOF | tee /etc/sysctl.d/k8s.conf
        
    2. 

  2. 

    
            net.ipv4.ip_forward = 1
        
    3. 

  3. 

    
            net.bridge.bridge-nf-call-ip6tables = 1
        
    4. 

  4. 

    
            net.bridge.bridge-nf-call-iptables = 1
        
    5. 

  5. 

    
            EOF
        
    6. 

  6. 

    
            sysctl -p /etc/sysctl.d/k8s.conf
        




报错提示不存在的话 安装这个模块


modprobe br_netfilter




1.4、安装 Docker




  1. 

    
            wget  https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
        
    2. 

  2. 

    
            mv docker-ce.repo  /etc/yum.repos.d/
        
    3. 

  3. 

    
            yum install docker-ce -y
        
    4. 

  4. 

    
            systemctl start docker && systemctl enable docker
        




1.5、创建安装目录




  1. 

    
            mkdir /k8s/etcd/{bin,cfg,ssl} -p
        
    2. 

  2. 

    
            mkdir /k8s/kubernetes/{bin,cfg,ssl} -p
        




1.6、安装及配置CFSSL




  1. 

    
            wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
        
    2. 

  2. 

    
            wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
        
    3. 

  3. 

    
            wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
        
    4. 

  4. 

    
            chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
        
    5. 

  5. 

    
            mv cfssl_linux-amd64 /usr/local/bin/cfssl
        
    6. 

  6. 

    
            mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
        
    7. 

  7. 

    
            mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
        




1.7、创建认证证书


创建 ETCD 证书




  1. 

    
            cat << EOF | tee ca-config.json
        
    2. 

  2. 

    
            {
        
    3. 

  3. 

    
            "signing": {
        
    4. 

  4. 

    
            "default": {
        
    5. 

  5. 

    
            "expiry": "87600h"
        
    6. 

  6. 

    
            },
        
    7. 

  7. 

    
            "profiles": {
        
    8. 

  8. 

    
            "www": {
        
    9. 

  9. 

    
            "expiry": "87600h",
        
    10. 

  10. 

    
            "usages": [
        
    11. 

  11. 

    
            "signing",
        
    12. 

  12. 

    
            "key encipherment",
        
    13. 

  13. 

    
            "server auth",
        
    14. 

  14. 

    
            "client auth"
        
    15. 

  15. 

    
            ]
        
    16. 

  16. 

    
            }
        
    17. 

  17. 

    
            }
        
    18. 

  18. 

    
            }
        
    19. 

  19. 

    
            }
        
    20. 

  20. 

    
            EOF
        




创建 ETCD CA 配置文件




  1. 

    
            cat << EOF | tee ca-csr.json
        
    2. 

  2. 

    
            {
        
    3. 

  3. 

    
            "CN": "etcd CA",
        
    4. 

  4. 

    
            "key": {
        
    5. 

  5. 

    
            "algo": "rsa",
        
    6. 

  6. 

    
            "size": 2048
        
    7. 

  7. 

    
            },
        
    8. 

  8. 

    
            "names": [
        
    9. 

  9. 

    
            {
        
    10. 

  10. 

    
            "C": "CN",
        
    11. 

  11. 

    
            "L": "Shenzhen",
        
    12. 

  12. 

    
            "ST": "Shenzhen"
        
    13. 

  13. 

    
            }
        
    14. 

  14. 

    
            ]
        
    15. 

  15. 

    
            }
        
    16. 

  16. 

    
            EOF
        




创建 ETCD Server 证书




  1. 

    
            cat << EOF | tee server-csr.json
        
    2. 

  2. 

    
            {
        
    3. 

  3. 

    
            "CN": "etcd",
        
    4. 

  4. 

    
            "hosts": [
        
    5. 

  5. 

    
            "10.0.0.1",
        
    6. 

  6. 

    
            "192.168.1.108",
        
    7. 

  7. 

    
            "192.168.1.165",
        
    8. 

  8. 

    
            "192.168.1.215"
        
    9. 

  9. 

    
            ],
        
    10. 

  10. 

    
            "key": {
        
    11. 

  11. 

    
            "algo": "rsa",
        
    12. 

  12. 

    
            "size": 2048
        
    13. 

  13. 

    
            },
        
    14. 

  14. 

    
            "names": [
        
    15. 

  15. 

    
            {
        
    16. 

  16. 

    
            "C": "CN",
        
    17. 

  17. 

    
            "L": "Shenzhen",
        
    18. 

  18. 

    
            "ST": "Shenzhen"
        
    19. 

  19. 

    
            }
        
    20. 

  20. 

    
            ]
        
    21. 

  21. 

    
            }
        
    22. 

  22. 

    
            EOF
        




生成 ETCD CA 证书和私钥




  1. 

    
            cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
        
    2. 

  2. 

    
            cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
        




创建 Kubernetes CA 证书




  1. 

    
            cat << EOF | tee ca-config.json
        
    2. 

  2. 

    
            {
        
    3. 

  3. 

    
            "signing": {
        
    4. 

  4. 

    
            "default": {
        
    5. 

  5. 

    
            "expiry": "87600h"
        
    6. 

  6. 

    
            },
        
    7. 

  7. 

    
            "profiles": {
        
    8. 

  8. 

    
            "kubernetes": {
        
    9. 

  9. 

    
            "expiry": "87600h",
        
    10. 

  10. 

    
            "usages": [
        
    11. 

  11. 

    
            "signing",
        
    12. 

  12. 

    
            "key encipherment",
        
    13. 

  13. 

    
            "server auth",
        
    14. 

  14. 

    
            "client auth"
        
    15. 

  15. 

    
            ]
        
    16. 

  16. 

    
            }
        
    17. 

  17. 

    
            }
        
    18. 

  18. 

    
            }
        
    19. 

  19. 

    
            }
        
    20. 

  20. 

    
            EOF
        






  1. 

    
            cat << EOF | tee ca-csr.json
        
    2. 

  2. 

    
            {
        
    3. 

  3. 

    
            "CN": "kubernetes",
        
    4. 

  4. 

    
            "key": {
        
    5. 

  5. 

    
            "algo": "rsa",
        
    6. 

  6. 

    
            "size": 2048
        
    7. 

  7. 

    
            },
        
    8. 

  8. 

    
            "names": [
        
    9. 

  9. 

    
            {
        
    10. 

  10. 

    
            "C": "CN",
        
    11. 

  11. 

    
            "L": "Shenzhen",
        
    12. 

  12. 

    
            "ST": "Shenzhen",
        
    13. 

  13. 

    
            "O": "k8s",
        
    14. 

  14. 

    
            "OU": "System"
        
    15. 

  15. 

    
            }
        
    16. 

  16. 

    
            ]
        
    17. 

  17. 

    
            }
        
    18. 

  18. 

    
            EOF
        
    19. 

  19. 

    20. 

  20. 





生成证书


cfssl gencert -initca ca-csr.json | cfssljson -bare ca -




生成API_SERVER证书




  1. 

    
            cat << EOF | tee server-csr.json
        
    2. 

  2. 

    
            {
        
    3. 

  3. 

    
            "CN": "kubernetes",
        
    4. 

  4. 

    
            "hosts": [
        
    5. 

  5. 

    
            "192.168.1.108",
        
    6. 

  6. 

    
            "192.168.1.165",
        
    7. 

  7. 

    
            "192.168.1.215",
        
    8. 

  8. 

    
            "kubernetes",
        
    9. 

  9. 

    
            "kubernetes.default",
        
    10. 

  10. 

    
            "kubernetes.default.svc",
        
    11. 

  11. 

    
            "kubernetes.default.svc.cluster",
        
    12. 

  12. 

    
            "kubernetes.default.svc.cluster.local"
        
    13. 

  13. 

    
            ],
        
    14. 

  14. 

    
            "key": {
        
    15. 

  15. 

    
            "algo": "rsa",
        
    16. 

  16. 

    
            "size": 2048
        
    17. 

  17. 

    
            },
        
    18. 

  18. 

    
            "names": [
        
    19. 

  19. 

    
            {
        
    20. 

  20. 

    
            "C": "CN",
        
    21. 

  21. 

    
            "L": "Shenzhen",
        
    22. 

  22. 

    
            "ST": "Shenzhen",
        
    23. 

  23. 

    
            "O": "k8s",
        
    24. 

  24. 

    
            "OU": "System"
        
    25. 

  25. 

    
            }
        
    26. 

  26. 

    
            ]
        
    27. 

  27. 

    
            }
        
    28. 

  28. 

    
            EOF
        
    29. 

  29. 





生成证书


cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server




创建 Kubernetes Proxy 证书




  1. 

    
            cat << EOF | tee kube-proxy-csr.json
        
    2. 

  2. 

    
            {
        
    3. 

  3. 

    
            "CN": "system:kube-proxy",
        
    4. 

  4. 

    
            "hosts": [],
        
    5. 

  5. 

    
            "key": {
        
    6. 

  6. 

    
            "algo": "rsa",
        
    7. 

  7. 

    
            "size": 2048
        
    8. 

  8. 

    
            },
        
    9. 

  9. 

    
            "names": [
        
    10. 

  10. 

    
            {
        
    11. 

  11. 

    
            "C": "CN",
        
    12. 

  12. 

    
            "L": "Shenzhen",
        
    13. 

  13. 

    
            "ST": "Shenzhen",
        
    14. 

  14. 

    
            "O": "k8s",
        
    15. 

  15. 

    
            "OU": "System"
        
    16. 

  16. 

    
            }
        
    17. 

  17. 

    
            ]
        
    18. 

  18. 

    
            }
        
    19. 

  19. 

    
            EOF
        
    20. 

  20. 

    21. 

  21. 

    
            cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
        




1.8、 ssh-key认证




  1. 

    
            # ssh-keygen
        
    2. 

  2. 

    
            Generating public/private rsa key pair.
        
    3. 

  3. 

    
            Enter file in which to save the key (/root/.ssh/id_rsa):
        
    4. 

  4. 

    
            Created directory '/root/.ssh'.
        
    5. 

  5. 

    
            Enter passphrase (empty for no passphrase):
        
    6. 

  6. 

    
            Enter same passphrase again:
        
    7. 

  7. 

    
            Your identification has been saved in /root/.ssh/id_rsa.
        
    8. 

  8. 

    
            Your public key has been saved in /root/.ssh/id_rsa.pub.
        
    9. 

  9. 

    
            The key fingerprint is:
        
    10. 

  10. 

    
            SHA256:FQjjiRDp8IKGT+UDM+GbQLBzF3DqDJ+pKnMIcHGyO/o root@qas-k8s-master01
        
    11. 

  11. 

    
            The key's randomart image is:
        
    12. 

  12. 

    
            +---[RSA 2048]----+
        
    13. 

  13. 

    
            |o.==o o. .. |
        
    14. 

  14. 

    
            |ooB+o+ o. . |
        
    15. 

  15. 

    
            |B++@o o . |
        
    16. 

  16. 

    
            |=X**o . |
        
    17. 

  17. 

    
            |o=O. . S |
        
    18. 

  18. 

    
            |..+ |
        
    19. 

  19. 

    
            |oo . |
        
    20. 

  20. 

    
            |* . |
        
    21. 

  21. 

    
            |o+E |
        
    22. 

  22. 

    
            +----[SHA256]-----+
        
    23. 

  23. 

    24. 

  24. 

    
            # ssh-copy-id 192.168.1.165
        
    25. 

  25. 

    
            # ssh-copy-id 192.168.1.215
        




2 、部署ETCD


安装包地址：


解压安装文件




  1. 

    
            tar -xvf etcd-v3.3.10-linux-amd64.tar.gz
        
    2. 

  2. 

    
            cd etcd-v3.3.10-linux-amd64/
        
    3. 

  3. 

    
            cp etcd etcdctl /k8s/etcd/bin/
        




设置etcd的配置文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/etcd/cfg/etcd
        
    2. 

  2. 

    
            #[Member]
        
    3. 

  3. 

    
            ETCD_NAME="etcd01"
        
    4. 

  4. 

    
            ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
        
    5. 

  5. 

    
            ETCD_LISTEN_PEER_URLS="https://192.168.1.108:2380"
        
    6. 

  6. 

    
            ETCD_LISTEN_CLIENT_URLS="https://192.168.1.108:2379"
        
    7. 

  7. 

    8. 

  8. 

    
            #[Clustering]
        
    9. 

  9. 

    
            ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.108:2380"
        
    10. 

  10. 

    
            ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.108:2379"
        
    11. 

  11. 

    
            ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.108:2380,etcd02=https://192.168.1.165:2380,etcd03=https://192.168.1.215:2380"
        
    12. 

  12. 

    
            ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
        
    13. 

  13. 

    
            ETCD_INITIAL_CLUSTER_STATE="new"
        
    14. 

  14. 

    
            [root@qas-k8s-master01 ssl]#
        




创建 etcd的 systemd unit 文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/etcd.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Etcd Server
        
    4. 

  4. 

    
            After=network.target
        
    5. 

  5. 

    
            After=network-online.target
        
    6. 

  6. 

    
            Wants=network-online.target
        
    7. 

  7. 

    8. 

  8. 

    
            [Service]
        
    9. 

  9. 

    
            Type=notify
        
    10. 

  10. 

    
            EnvironmentFile=/k8s/etcd/cfg/etcd
        
    11. 

  11. 

    
            ExecStart=/k8s/etcd/bin/etcd \
        
    12. 

  12. 

    
            --name=${ETCD_NAME} \
        
    13. 

  13. 

    
            --data-dir=${ETCD_DATA_DIR} \
        
    14. 

  14. 

    
            --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
        
    15. 

  15. 

    
            --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
        
    16. 

  16. 

    
            --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
        
    17. 

  17. 

    
            --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
        
    18. 

  18. 

    
            --initial-cluster=${ETCD_INITIAL_CLUSTER} \
        
    19. 

  19. 

    
            --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
        
    20. 

  20. 

    
            --initial-cluster-state=new \
        
    21. 

  21. 

    
            --cert-file=/k8s/etcd/ssl/server.pem \
        
    22. 

  22. 

    
            --key-file=/k8s/etcd/ssl/server-key.pem \
        
    23. 

  23. 

    
            --peer-cert-file=/k8s/etcd/ssl/server.pem \
        
    24. 

  24. 

    
            --peer-key-file=/k8s/etcd/ssl/server-key.pem \
        
    25. 

  25. 

    
            --trusted-ca-file=/k8s/etcd/ssl/ca.pem \
        
    26. 

  26. 

    
            --peer-trusted-ca-file=/k8s/etcd/ssl/ca.pem
        
    27. 

  27. 

    
            Restart=on-failure
        
    28. 

  28. 

    
            LimitNOFILE=65536
        
    29. 

  29. 

    30. 

  30. 

    
            [Install]
        
    31. 

  31. 

    
            WantedBy=multi-user.target
        
    32. 

  32. 

    
            [root@qas-k8s-master01 ssl]#
        




拷贝证书文件


cp ca*pem server*pem /k8s/etcd/ssl




启动ETCD服务




  1. 

    
            systemctl daemon-reload
        
    2. 

  2. 

    
            systemctl enable etcd
        
    3. 

  3. 

    
            systemctl start etcd
        




查看etcd状态




  1. 

    
            [root@qas-k8s-master01 ssl]# systemctl status etcd
        
    2. 

  2. 

    
            ● etcd.service - Etcd Server
        
    3. 

  3. 

    
            Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
        
    4. 

  4. 

    
            Active: active (running) since Tue 2019-07-09 15:55:58 CST; 1h 12min ago
        
    5. 

  5. 

    
            Main PID: 29830 (etcd)
        
    6. 

  6. 

    
            Tasks: 12
        
    7. 

  7. 

    
            Memory: 66.7M
        
    8. 

  8. 

    
            CGroup: /system.slice/etcd.service
        
    9. 

  9. 

    
            └─29830 /k8s/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://192.168.1.10...
        
    10. 

  10. 

    11. 

  11. 

    
            Jul 09 17:07:39 qas-k8s-master01 etcd[29830]: the clock difference against peer 2369b1ebd5f356e7 is too high [1.340843978s ...SAGE")
        
    12. 

  12. 

    
            Jul 09 17:07:39 qas-k8s-master01 etcd[29830]: the clock difference against peer 2369b1ebd5f356e7 is too high [1.343013157s ...SHOT")
        
    13. 

  13. 

    
            Jul 09 17:08:04 qas-k8s-master01 etcd[29830]: the clock difference against peer 67ad1178efd89256 is too high [1.287315582s ...SHOT")
        
    14. 

  14. 

    
            Jul 09 17:08:04 qas-k8s-master01 etcd[29830]: the clock difference against peer 67ad1178efd89256 is too high [1.285244104s ...SAGE")
        
    15. 

  15. 

    
            Jul 09 17:08:09 qas-k8s-master01 etcd[29830]: the clock difference against peer 2369b1ebd5f356e7 is too high [1.340602239s ...SAGE")
        
    16. 

  16. 

    
            Jul 09 17:08:09 qas-k8s-master01 etcd[29830]: the clock difference against peer 2369b1ebd5f356e7 is too high [1.342891162s ...SHOT")
        
    17. 

  17. 

    
            Jul 09 17:08:34 qas-k8s-master01 etcd[29830]: the clock difference against peer 67ad1178efd89256 is too high [1.285259365s ...SAGE")
        
    18. 

  18. 

    
            Jul 09 17:08:34 qas-k8s-master01 etcd[29830]: the clock difference against peer 67ad1178efd89256 is too high [1.287337806s ...SHOT")
        
    19. 

  19. 

    
            Jul 09 17:08:39 qas-k8s-master01 etcd[29830]: the clock difference against peer 2369b1ebd5f356e7 is too high [1.342999378s ...SHOT")
        
    20. 

  20. 

    
            Jul 09 17:08:39 qas-k8s-master01 etcd[29830]: the clock difference against peer 2369b1ebd5f356e7 is too high [1.339802949s ...SAGE")
        
    21. 

  21. 

    
            Hint: Some lines were ellipsized, use -l to show in full.
        
    22. 

  22. 

    
            [root@qas-k8s-master01 ssl]#
        




将启动文件、配置文件拷贝到 节点1、节点2




  1. 

    
            cd /k8s/
        
    2. 

  2. 

    
            scp -r etcd 192.168.1.165:/k8s/
        
    3. 

  3. 

    
            scp -r etcd 192.168.1.215:/k8s/
        
    4. 

  4. 

    
            scp /usr/lib/systemd/system/etcd.service 192.168.1.165:/usr/lib/systemd/system/etcd.service
        
    5. 

  5. 

    
            scp /usr/lib/systemd/system/etcd.service 192.168.1.215:/usr/lib/systemd/system/etcd.service
        




node01节点修改




  1. 

    
            [root@qas-k8s-node01 cfg]# cat /k8s/etcd/cfg/etcd
        
    2. 

  2. 

    
            #[Member]
        
    3. 

  3. 

    
            ETCD_NAME="etcd02"
        
    4. 

  4. 

    
            ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
        
    5. 

  5. 

    
            ETCD_LISTEN_PEER_URLS="https://192.168.1.165:2380"
        
    6. 

  6. 

    
            ETCD_LISTEN_CLIENT_URLS="https://192.168.1.165:2379"
        
    7. 

  7. 

    8. 

  8. 

    
            #[Clustering]
        
    9. 

  9. 

    
            ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.165:2380"
        
    10. 

  10. 

    
            ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.165:2379"
        
    11. 

  11. 

    
            ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.108:2380,etcd02=https://192.168.1.165:2380,etcd03=https://192.168.1.215:2380"
        
    12. 

  12. 

    
            ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
        
    13. 

  13. 

    
            ETCD_INITIAL_CLUSTER_STATE="new"
        
    14. 

  14. 

    
            [root@qas-k8s-node01 cfg]#
        




node02节点修改




  1. 

    
            [root@qas-k8s-node02 bin]# cat /k8s/etcd/cfg/etcd
        
    2. 

  2. 

    
            #[Member]
        
    3. 

  3. 

    
            ETCD_NAME="etcd03"
        
    4. 

  4. 

    
            ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
        
    5. 

  5. 

    
            ETCD_LISTEN_PEER_URLS="https://192.168.1.215:2380"
        
    6. 

  6. 

    
            ETCD_LISTEN_CLIENT_URLS="https://192.168.1.215:2379"
        
    7. 

  7. 

    8. 

  8. 

    
            #[Clustering]
        
    9. 

  9. 

    
            ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.215:2380"
        
    10. 

  10. 

    
            ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.215:2379"
        
    11. 

  11. 

    
            ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.108:2380,etcd02=https://192.168.1.165:2380,etcd03=https://192.168.1.215:2380"
        
    12. 

  12. 

    
            ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
        
    13. 

  13. 

    
            ETCD_INITIAL_CLUSTER_STATE="new"
        
    14. 

  14. 

    
            [root@qas-k8s-node02 bin]#
        




验证集群是否正常运行




  1. 

    
            ./etcdctl \
        
    2. 

  2. 

    
            --ca-file=/k8s/etcd/ssl/ca.pem \
        
    3. 

  3. 

    
            --cert-file=/k8s/etcd/ssl/server.pem \
        
    4. 

  4. 

    
            --key-file=/k8s/etcd/ssl/server-key.pem \
        
    5. 

  5. 

    
            --endpoints="https://192.168.1.108,\
        
    6. 

  6. 

    
            https://192.168.1.165:2379,\
        
    7. 

  7. 

    
            https://192.168.1.215:2379" cluster-health
        
    8. 

  8. 

    9. 

  9. 

    
            注意：
        
    10. 

  10. 

    
            启动ETCD集群同时启动二个节点，启动一个节点集群是无法正常启动的；
        




返回以下表示ok




  1. 

    
            member 5db3ea816863435 is healthy: got healthy result from https://192.168.1.108:2379
        
    2. 

  2. 

    
            member 991b5845cecb31b is healthy: got healthy result from https://192.168.1.165:2379
        
    3. 

  3. 

    
            member c67ee2780d64a0d4 is healthy: got healthy result from https://192.168.1.215:2379
        
    4. 

  4. 

    
            cluster is healthy
        




3、部署Flannel网络


向 etcd 写入集群 Pod 网段信息




  1. 

    
            cd /k8s/etcd/ssl/
        
    2. 

  2. 

    
            /k8s/etcd/bin/etcdctl \
        
    3. 

  3. 

    
            --ca-file=ca.pem --cert-file=server.pem \
        
    4. 

  4. 

    
            --key-file=server-key.pem \
        
    5. 

  5. 

    
            --endpoints="https://192.168.1.108:2379,\
        
    6. 

  6. 

    
            https://192.168.1.165:2379,https://192.168.1.215:2379" \
        
    7. 

  7. 

    
            set /coreos.com/network/config '{ "Network": "172.18.0.0/16", "Backend": {"Type": "vxlan"}}'
        




  • flanneld 当前版本 (v0.10.0) 不支持 etcd v3，故使用 etcd v2 API 写入配置 key 和网段数据；
    • 

  • 写入的 Pod 网段 ${CLUSTER_CIDR} 必须是 /16 段地址，必须与 kube-controller-manager 的 –cluster-cidr 参数值一致；


解压安装




  1. 

    
            tar -xvf flannel-v0.10.0-linux-amd64.tar.gz
        
    2. 

  2. 

    
            mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/
        




配置Flannel




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/kubernetes/cfg/flanneld
        
    2. 

  2. 

    
            FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.1.108:2379,https://192.168.1.165:2379,https://192.168.1.215:2379 -etcd-cafile=/k8s/etcd/ssl/ca.pem -etcd-certfile=/k8s/etcd/ssl/server.pem -etcd-keyfile=/k8s/etcd/ssl/server-key.pem"
        
    3. 

  3. 

    
            [root@qas-k8s-master01 ssl]#
        




创建 flanneld 的 systemd unit 文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/flanneld.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Flanneld overlay address etcd agent
        
    4. 

  4. 

    
            After=network-online.target network.target
        
    5. 

  5. 

    
            Before=docker.service
        
    6. 

  6. 

    7. 

  7. 

    
            [Service]
        
    8. 

  8. 

    
            Type=notify
        
    9. 

  9. 

    
            EnvironmentFile=/k8s/kubernetes/cfg/flanneld
        
    10. 

  10. 

    
            ExecStart=/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
        
    11. 

  11. 

    
            ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
        
    12. 

  12. 

    
            Restart=on-failure
        
    13. 

  13. 

    14. 

  14. 

    
            [Install]
        
    15. 

  15. 

    
            WantedBy=multi-user.target
        
    16. 

  16. 

    
            [root@qas-k8s-master01 ssl]#
        




  • mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件，后续 docker 启动时 使用这个文件中的环境变量配置 docker0 网桥；
    • 

  • flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 -iface 参数指定通信接口，如上面的 eth0 接口;
    • 

  • flanneld 运行时需要 root 权限；


配置Docker启动指定子网段




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/docker.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Docker Application Container Engine
        
    4. 

  4. 

    
            Documentation=https://docs.docker.com
        
    5. 

  5. 

    
            After=network-online.target firewalld.service
        
    6. 

  6. 

    
            Wants=network-online.target
        
    7. 

  7. 

    8. 

  8. 

    
            [Service]
        
    9. 

  9. 

    
            Type=notify
        
    10. 

  10. 

    
            EnvironmentFile=/run/flannel/subnet.env
        
    11. 

  11. 

    
            ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
        
    12. 

  12. 

    
            ExecReload=/bin/kill -s HUP $MAINPID
        
    13. 

  13. 

    
            LimitNOFILE=infinity
        
    14. 

  14. 

    
            LimitNPROC=infinity
        
    15. 

  15. 

    
            LimitCORE=infinity
        
    16. 

  16. 

    
            TimeoutStartSec=0
        
    17. 

  17. 

    
            Delegate=yes
        
    18. 

  18. 

    
            KillMode=process
        
    19. 

  19. 

    
            Restart=on-failure
        
    20. 

  20. 

    
            StartLimitBurst=3
        
    21. 

  21. 

    
            StartLimitInterval=60s
        
    22. 

  22. 

    23. 

  23. 

    
            [Install]
        
    24. 

  24. 

    
            WantedBy=multi-user.target
        
    25. 

  25. 

    
            [root@qas-k8s-master01 ssl]#
        




将flanneld systemd unit 文件到所有节点




  1. 

    
            cd /k8s/
        
    2. 

  2. 

    
            scp -r kubernetes 192.168.1.165:/k8s/
        
    3. 

  3. 

    
            scp -r kubernetes 192.168.1.215:/k8s/
        
    4. 

  4. 

    
            scp /k8s/kubernetes/cfg/flanneld 192.168.1.165:/k8s/kubernetes/cfg/flanneld
        
    5. 

  5. 

    
            scp /k8s/kubernetes/cfg/flanneld 192.168.1.215:/k8s/kubernetes/cfg/flanneld
        
    6. 

  6. 

    
            scp /usr/lib/systemd/system/docker.service 192.168.1.165:/usr/lib/systemd/system/docker.service
        
    7. 

  7. 

    
            scp /usr/lib/systemd/system/docker.service 192.168.1.215:/usr/lib/systemd/system/docker.service
        
    8. 

  8. 

    
            scp /usr/lib/systemd/system/flanneld.service 192.168.1.165:/usr/lib/systemd/system/flanneld.service
        
    9. 

  9. 

    
            scp /usr/lib/systemd/system/flanneld.service 192.168.1.215:/usr/lib/systemd/system/flanneld.service
        
    10. 

  10. 





启动服务




  1. 

    
            systemctl daemon-reload
        
    2. 

  2. 

    
            systemctl start flanneld
        
    3. 

  3. 

    
            systemctl enable flanneld
        
    4. 

  4. 

    
            systemctl restart docker
        




查看node01服务




  1. 

    
            [root@qas-k8s-node01 cfg]# systemctl status flanneld
        
    2. 

  2. 

    
            ● flanneld.service - Flanneld overlay address etcd agent
        
    3. 

  3. 

    
            Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
        
    4. 

  4. 

    
            Active: active (running) since Tue 2019-07-09 14:51:52 +08; 2h 4min ago
        
    5. 

  5. 

    
            Main PID: 2191 (flanneld)
        
    6. 

  6. 

    
            Tasks: 10
        
    7. 

  7. 

    
            Memory: 14.8M
        
    8. 

  8. 

    
            CGroup: /system.slice/flanneld.service
        
    9. 

  9. 

    
            └─2191 /k8s/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://192.168.1.108:2379,https://192.168.1.165:2379,h...
        
    10. 

  10. 

    11. 

  11. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.577551 2191 iptables.go:167] Deleting iptables rule: ! -s ...RETURN
        
    12. 

  12. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.578182 2191 iptables.go:167] Deleting iptables rule: ! -s ...UERADE
        
    13. 

  13. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.578982 2191 iptables.go:167] Deleting iptables rule: -d 17...ACCEPT
        
    14. 

  14. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.579680 2191 iptables.go:155] Adding iptables rule: -s 172....ACCEPT
        
    15. 

  15. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.580578 2191 iptables.go:155] Adding iptables rule: -s 172....RETURN
        
    16. 

  16. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.582778 2191 iptables.go:155] Adding iptables rule: -s 172....UERADE
        
    17. 

  17. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.583542 2191 iptables.go:155] Adding iptables rule: -d 172....ACCEPT
        
    18. 

  18. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.584722 2191 iptables.go:155] Adding iptables rule: ! -s 17...RETURN
        
    19. 

  19. 

    
            Jul 09 14:51:52 qas-k8s-node01 flanneld[2191]: I0709 14:51:52.587207 2191 iptables.go:155] Adding iptables rule: ! -s 17...UERADE
        
    20. 

  20. 

    
            Jul 09 14:51:52 qas-k8s-node01 systemd[1]: Started Flanneld overlay address etcd agent.
        
    21. 

  21. 

    
            Hint: Some lines were ellipsized, use -l to show in full.
        
    22. 

  22. 

    
            [root@qas-k8s-node01 cfg]#
        




查看node02服务




  1. 

    
            [root@qas-k8s-node02 bin]# systemctl status flanneld
        
    2. 

  2. 

    
            ● flanneld.service - Flanneld overlay address etcd agent
        
    3. 

  3. 

    
            Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
        
    4. 

  4. 

    
            Active: active (running) since Tue 2019-07-09 14:51:57 CST; 2h 5min ago
        
    5. 

  5. 

    
            Main PID: 12986 (flanneld)
        
    6. 

  6. 

    
            Tasks: 11
        
    7. 

  7. 

    
            Memory: 10.2M
        
    8. 

  8. 

    
            CGroup: /system.slice/flanneld.service
        
    9. 

  9. 

    
            └─12986 /k8s/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://192.168.1.108:2379,https://192...
        
    10. 

  10. 

    11. 

  11. 

    
            Jul 09 14:51:56 qas-k8s-node02 flanneld[12986]: I0709 14:51:56.992347 12986 iptables.go:167] Deleting iptab...CEPT
        
    12. 

  12. 

    
            Jul 09 14:51:56 qas-k8s-node02 flanneld[12986]: I0709 14:51:56.992940 12986 iptables.go:155] Adding iptable...TURN
        
    13. 

  13. 

    
            Jul 09 14:51:56 qas-k8s-node02 flanneld[12986]: I0709 14:51:56.994247 12986 iptables.go:155] Adding iptable...RADE
        
    14. 

  14. 

    
            Jul 09 14:51:56 qas-k8s-node02 flanneld[12986]: I0709 14:51:56.995163 12986 iptables.go:167] Deleting iptab...CEPT
        
    15. 

  15. 

    
            Jul 09 14:51:56 qas-k8s-node02 flanneld[12986]: I0709 14:51:56.995931 12986 main.go:429] Waiting for 22h59m...ease
        
    16. 

  16. 

    
            Jul 09 14:51:57 qas-k8s-node02 flanneld[12986]: I0709 14:51:56.999655 12986 iptables.go:155] Adding iptable...TURN
        
    17. 

  17. 

    
            Jul 09 14:51:57 qas-k8s-node02 flanneld[12986]: I0709 14:51:57.000668 12986 iptables.go:155] Adding iptable...CEPT
        
    18. 

  18. 

    
            Jul 09 14:51:57 qas-k8s-node02 flanneld[12986]: I0709 14:51:57.004930 12986 iptables.go:155] Adding iptable...RADE
        
    19. 

  19. 

    
            Jul 09 14:51:57 qas-k8s-node02 flanneld[12986]: I0709 14:51:57.005116 12986 iptables.go:155] Adding iptable...CEPT
        
    20. 

  20. 

    
            Jul 09 14:51:57 qas-k8s-node02 systemd[1]: Started Flanneld overlay address etcd agent.
        
    21. 

  21. 

    
            Hint: Some lines were ellipsized, use -l to show in full.
        




查看是否生效




  1. 

    
            [root@qas-k8s-node02 bin]# ip add
        
    2. 

  2. 

    
            1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        
    3. 

  3. 

    
            link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        
    4. 

  4. 

    
            inet 127.0.0.1/8 scope host lo
        
    5. 

  5. 

    
            valid_lft forever preferred_lft forever
        
    6. 

  6. 

    
            inet6 ::1/128 scope host
        
    7. 

  7. 

    
            valid_lft forever preferred_lft forever
        
    8. 

  8. 

    
            2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        
    9. 

  9. 

    
            link/ether 08:00:27:a0:69:83 brd ff:ff:ff:ff:ff:ff
        
    10. 

  10. 

    
            inet 192.168.1.215/24 brd 192.168.1.255 scope global noprefixroute enp0s3
        
    11. 

  11. 

    
            valid_lft forever preferred_lft forever
        
    12. 

  12. 

    
            inet6 fe80::1fd3:aade:96a5:e5b4/64 scope link noprefixroute
        
    13. 

  13. 

    
            valid_lft forever preferred_lft forever
        
    14. 

  14. 

    
            3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
        
    15. 

  15. 

    
            link/ether 02:42:68:7c:ef:e8 brd ff:ff:ff:ff:ff:ff
        
    16. 

  16. 

    
            inet 172.18.88.1/24 brd 172.18.88.255 scope global docker0
        
    17. 

  17. 

    
            valid_lft forever preferred_lft forever
        
    18. 

  18. 

    
            4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
        
    19. 

  19. 

    
            link/ether ee:7c:89:a4:9e:20 brd ff:ff:ff:ff:ff:ff
        
    20. 

  20. 

    
            inet 172.18.88.0/32 scope global flannel.1
        
    21. 

  21. 

    
            valid_lft forever preferred_lft forever
        
    22. 

  22. 

    
            inet6 fe80::ec7c:89ff:fea4:9e20/64 scope link
        
    23. 

  23. 

    
            valid_lft forever preferred_lft forever
        
    24. 

  24. 

    
            [root@qas-k8s-node02 bin]#
        




4、部署 master 节点


kubernetes master 节点运行如下组件：


  • kube-apiserver
    • 

  • kube-scheduler
    • 

  • kube-controller-manager
    
        kube-scheduler 和 kube-controller-manager 可以以集群模式运行，通过 leader 选举产生一个工作进程，其它进程处于阻塞模式。


将二进制文件解压拷贝到master 节点




  1. 

    
            tar -xvf kubernetes-server-linux-amd64.tar.gz
        
    2. 

  2. 

    
            cd kubernetes/server/bin/
        
    3. 

  3. 

    
            cp kube-scheduler kube-apiserver kube-controller-manager kubectl /k8s/kubernetes/bin/
        




拷贝认证


cp *pem /k8s/kubernetes/ssl/




部署 kube-apiserver 组件


创建 TLS Bootstrapping Token




  1. 

    
            # head -c 16 /dev/urandom | od -An -t x | tr -d ' '
        
    2. 

  2. 

    
            2366a641f656a0a025abb4aabda4511b
        
    3. 

  3. 

    4. 

  4. 

    
            vim /k8s/kubernetes/cfg/token.csv
        
    5. 

  5. 

    
            2366a641f656a0a025abb4aabda4511b,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
        




创建apiserver配置文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/kubernetes/cfg/kube-apiserver
        
    2. 

  2. 

    
            KUBE_APISERVER_OPTS="--logtostderr=true \
        
    3. 

  3. 

    
            --v=4 \
        
    4. 

  4. 

    
            --etcd-servers=https://192.168.1.108:2379,https://192.168.1.165:2379,https://192.168.1.215:2379 \
        
    5. 

  5. 

    
            --bind-address=192.168.1.108 \
        
    6. 

  6. 

    
            --secure-port=6443 \
        
    7. 

  7. 

    
            --advertise-address=192.168.1.108 \
        
    8. 

  8. 

    
            --allow-privileged=true \
        
    9. 

  9. 

    
            --service-cluster-ip-range=10.0.0.0/24 \
        
    10. 

  10. 

    
            --enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
        
    11. 

  11. 

    
            --authorization-mode=RBAC,Node \
        
    12. 

  12. 

    
            --enable-bootstrap-token-auth \
        
    13. 

  13. 

    
            --token-auth-file=/k8s/kubernetes/cfg/token.csv \
        
    14. 

  14. 

    
            --service-node-port-range=30000-50000 \
        
    15. 

  15. 

    
            --tls-cert-file=/k8s/kubernetes/ssl/server.pem \
        
    16. 

  16. 

    
            --tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem \
        
    17. 

  17. 

    
            --client-ca-file=/k8s/kubernetes/ssl/ca.pem \
        
    18. 

  18. 

    
            --service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem \
        
    19. 

  19. 

    
            --etcd-cafile=/k8s/etcd/ssl/ca.pem \
        
    20. 

  20. 

    
            --etcd-certfile=/k8s/etcd/ssl/server.pem \
        
    21. 

  21. 

    
            --etcd-keyfile=/k8s/etcd/ssl/server-key.pem"
        
    22. 

  22. 

    
            [root@qas-k8s-master01 ssl]#
        




创建 kube-apiserver systemd unit 文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/kube-apiserver.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Kubernetes API Server
        
    4. 

  4. 

    
            Documentation=https://github.com/kubernetes/kubernetes
        
    5. 

  5. 

    6. 

  6. 

    
            [Service]
        
    7. 

  7. 

    
            EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver
        
    8. 

  8. 

    
            ExecStart=/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
        
    9. 

  9. 

    
            Restart=on-failure
        
    10. 

  10. 

    11. 

  11. 

    
            [Install]
        
    12. 

  12. 

    
            WantedBy=multi-user.target
        
    13. 

  13. 

    
            [root@qas-k8s-master01 ssl]#
        




启动服务




  1. 

    
            systemctl daemon-reload
        
    2. 

  2. 

    
            systemctl enable kube-apiserver
        
    3. 

  3. 

    
            systemctl restart kube-apiserver
        




查看apiserver是否运行




  1. 

    
            [root@qas-k8s-master01 ssl]# systemctl status kube-apiserver
        
    2. 

  2. 

    
            ● kube-apiserver.service - Kubernetes API Server
        
    3. 

  3. 

    
            Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
        
    4. 

  4. 

    
            Active: active (running) since Tue 2019-07-09 15:56:17 CST; 1h 5min ago
        
    5. 

  5. 

    
            Docs: https://github.com/kubernetes/kubernetes
        
    6. 

  6. 

    
            Main PID: 29887 (kube-apiserver)
        
    7. 

  7. 

    
            Tasks: 11
        
    8. 

  8. 

    
            Memory: 294.0M
        
    9. 

  9. 

    
            CGroup: /system.slice/kube-apiserver.service
        
    10. 

  10. 

    
            └─29887 /k8s/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --etcd-servers=https://192.168.1.108:2379,https://...
        
    11. 

  11. 

    12. 

  12. 

    
            Jul 09 17:01:24 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:24.393003 29887 wrap.go:47] GET /api/v1/namespaces/def...6070]
        
    13. 

  13. 

    
            Jul 09 17:01:24 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:24.429849 29887 wrap.go:47] GET /api/v1/namespaces/kub...0182]
        
    14. 

  14. 

    
            Jul 09 17:01:24 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:24.433622 29887 wrap.go:47] PUT /api/v1/namespaces/kub...0182]
        
    15. 

  15. 

    
            Jul 09 17:01:25 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:25.392111 29887 wrap.go:47] GET /api/v1/namespaces/kub...1694]
        
    16. 

  16. 

    
            Jul 09 17:01:25 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:25.399618 29887 wrap.go:47] PUT /api/v1/namespaces/kub...1694]
        
    17. 

  17. 

    
            Jul 09 17:01:26 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:26.270688 29887 wrap.go:47] GET /apis/coordination.k8s...3914]
        
    18. 

  18. 

    
            Jul 09 17:01:26 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:26.275799 29887 wrap.go:47] PUT /apis/coordination.k8s...3914]
        
    19. 

  19. 

    
            Jul 09 17:01:26 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:26.283494 29887 wrap.go:47] GET /api/v1/nodes: (2.0970...0182]
        
    20. 

  20. 

    
            Jul 09 17:01:26 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:26.461482 29887 wrap.go:47] GET /api/v1/namespaces/kub...0182]
        
    21. 

  21. 

    
            Jul 09 17:01:26 qas-k8s-master01 kube-apiserver[29887]: I0709 17:01:26.466095 29887 wrap.go:47] PUT /api/v1/namespaces/kub...0182]
        
    22. 

  22. 

    
            Hint: Some lines were ellipsized, use -l to show in full.
        
    23. 

  23. 

    
            [root@qas-k8s-master01 ssl]#
        




部署kube-scheduler


创建kube-scheduler配置文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/kubernetes/cfg/kube-scheduler
        
    2. 

  2. 

    
            KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect"
        
    3. 

  3. 

    
            [root@qas-k8s-master01 ssl]#
        




创建kube-scheduler systemd unit 文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/kube-scheduler.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Kubernetes Scheduler
        
    4. 

  4. 

    
            Documentation=https://github.com/kubernetes/kubernetes
        
    5. 

  5. 

    6. 

  6. 

    
            [Service]
        
    7. 

  7. 

    
            EnvironmentFile=-/k8s/kubernetes/cfg/kube-scheduler
        
    8. 

  8. 

    
            ExecStart=/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
        
    9. 

  9. 

    
            Restart=on-failure
        
    10. 

  10. 

    11. 

  11. 

    
            [Install]
        
    12. 

  12. 

    
            WantedBy=multi-user.target
        
    13. 

  13. 

    
            [root@qas-k8s-master01 ssl]#
        




启动服务




  1. 

    
            systemctl daemon-reload
        
    2. 

  2. 

    
            systemctl enable kube-scheduler.service
        
    3. 

  3. 

    
            systemctl restart kube-scheduler.service
        




查看kube-scheduler是否运行




  1. 

    
            [root@qas-k8s-master01 ssl]# systemctl status kube-scheduler.service
        
    2. 

  2. 

    
            ● kube-scheduler.service - Kubernetes Scheduler
        
    3. 

  3. 

    
            Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
        
    4. 

  4. 

    
            Active: active (running) since Tue 2019-07-09 15:56:35 CST; 1h 7min ago
        
    5. 

  5. 

    
            Docs: https://github.com/kubernetes/kubernetes
        
    6. 

  6. 

    
            Main PID: 29993 (kube-scheduler)
        
    7. 

  7. 

    
            Tasks: 10
        
    8. 

  8. 

    
            Memory: 43.1M
        
    9. 

  9. 

    
            CGroup: /system.slice/kube-scheduler.service
        
    10. 

  10. 

    
            └─29993 /k8s/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect
        
    11. 

  11. 

    12. 

  12. 

    
            Jul 09 16:56:04 qas-k8s-master01 kube-scheduler[29993]: I0709 16:56:04.018734 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    13. 

  13. 

    
            Jul 09 16:56:39 qas-k8s-master01 kube-scheduler[29993]: I0709 16:56:39.010296 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    14. 

  14. 

    
            Jul 09 16:58:58 qas-k8s-master01 kube-scheduler[29993]: I0709 16:58:58.041986 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    15. 

  15. 

    
            Jul 09 16:59:43 qas-k8s-master01 kube-scheduler[29993]: I0709 16:59:43.994356 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    16. 

  16. 

    
            Jul 09 17:00:24 qas-k8s-master01 kube-scheduler[29993]: I0709 17:00:24.022221 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    17. 

  17. 

    
            Jul 09 17:00:26 qas-k8s-master01 kube-scheduler[29993]: I0709 17:00:26.001668 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    18. 

  18. 

    
            Jul 09 17:00:45 qas-k8s-master01 kube-scheduler[29993]: I0709 17:00:45.012870 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    19. 

  19. 

    
            Jul 09 17:02:49 qas-k8s-master01 kube-scheduler[29993]: I0709 17:02:49.017949 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    20. 

  20. 

    
            Jul 09 17:03:00 qas-k8s-master01 kube-scheduler[29993]: I0709 17:03:00.021114 29993 reflector.go:385] k8s.io/kubernetes/cm...eived
        
    21. 

  21. 

    
            Jul 09 17:03:20 qas-k8s-master01 kube-scheduler[29993]: I0709 17:03:20.002870 29993 reflector.go:385] k8s.io/client-go/inf...eived
        
    22. 

  22. 

    
            Hint: Some lines were ellipsized, use -l to show in full.
        
    23. 

  23. 

    
            [root@qas-k8s-master01 ssl]#
        




部署kube-controller-manager


创建kube-controller-manager配置文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/kubernetes/cfg/kube-controller-manager
        
    2. 

  2. 

    
            KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
        
    3. 

  3. 

    
            --v=4 \
        
    4. 

  4. 

    
            --master=127.0.0.1:8080 \
        
    5. 

  5. 

    
            --leader-elect=true \
        
    6. 

  6. 

    
            --address=127.0.0.1 \
        
    7. 

  7. 

    
            --service-cluster-ip-range=10.0.0.0/24 \
        
    8. 

  8. 

    
            --cluster-name=kubernetes \
        
    9. 

  9. 

    
            --cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem \
        
    10. 

  10. 

    
            --cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem \
        
    11. 

  11. 

    
            --root-ca-file=/k8s/kubernetes/ssl/ca.pem \
        
    12. 

  12. 

    
            --service-account-private-key-file=/k8s/kubernetes/ssl/ca-key.pem"
        
    13. 

  13. 

    
            [root@qas-k8s-master01 ssl]#
        




创建kube-controller-manager systemd unit 文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/kube-controller-manager.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Kubernetes Controller Manager
        
    4. 

  4. 

    
            Documentation=https://github.com/kubernetes/kubernetes
        
    5. 

  5. 

    6. 

  6. 

    
            [Service]
        
    7. 

  7. 

    
            EnvironmentFile=-/k8s/kubernetes/cfg/kube-controller-manager
        
    8. 

  8. 

    
            ExecStart=/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
        
    9. 

  9. 

    
            Restart=on-failure
        
    10. 

  10. 

    11. 

  11. 

    
            [Install]
        
    12. 

  12. 

    
            WantedBy=multi-user.target
        
    13. 

  13. 

    
            [root@qas-k8s-master01 ssl]#
        




启动服务




  1. 

    
            systemctl daemon-reload
        
    2. 

  2. 

    
            systemctl enable kube-controller-manager
        
    3. 

  3. 

    
            systemctl restart kube-controller-manager
        




查看kube-controller-manager是否运行




  1. 

    
            [root@qas-k8s-master01 ssl]# systemctl status kube-controller-manager
        
    2. 

  2. 

    
            ● kube-controller-manager.service - Kubernetes Controller Manager
        
    3. 

  3. 

    
            Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
        
    4. 

  4. 

    
            Active: active (running) since Tue 2019-07-09 15:56:45 CST; 1h 8min ago
        
    5. 

  5. 

    
            Docs: https://github.com/kubernetes/kubernetes
        
    6. 

  6. 

    
            Main PID: 30021 (kube-controller)
        
    7. 

  7. 

    
            Tasks: 9
        
    8. 

  8. 

    
            Memory: 132.4M
        
    9. 

  9. 

    
            CGroup: /system.slice/kube-controller-manager.service
        
    10. 

  10. 

    
            └─30021 /k8s/kubernetes/bin/kube-controller-manager --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect=tr...
        
    11. 

  11. 

    12. 

  12. 

    
            Jul 09 17:05:04 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:04.041136 30021 request.go:530] Throttling req...=32s
        
    13. 

  13. 

    
            Jul 09 17:05:04 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:04.085887 30021 request.go:530] Throttling req...=32s
        
    14. 

  14. 

    
            Jul 09 17:05:04 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:04.961864 30021 reflector.go:243] k8s.io/clien...sync
        
    15. 

  15. 

    
            Jul 09 17:05:05 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:05.038589 30021 reflector.go:243] k8s.io/clien...sync
        
    16. 

  16. 

    
            Jul 09 17:05:05 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:05.421412 30021 reflector.go:243] k8s.io/clien...sync
        
    17. 

  17. 

    
            Jul 09 17:05:05 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:05.564957 30021 pv_controller_base.go:419] res...ller
        
    18. 

  18. 

    
            Jul 09 17:05:06 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:06.380376 30021 gc_controller.go:144] GC'ing orphaned
        
    19. 

  19. 

    
            Jul 09 17:05:06 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:06.383448 30021 gc_controller.go:173] GC'ing u...ing.
        
    20. 

  20. 

    
            Jul 09 17:05:07 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:07.419696 30021 cronjob_controller.go:128] Fou...jobs
        
    21. 

  21. 

    
            Jul 09 17:05:07 qas-k8s-master01 kube-controller-manager[30021]: I0709 17:05:07.419707 30021 cronjob_controller.go:134] Fou...oups
        
    22. 

  22. 

    
            Hint: Some lines were ellipsized, use -l to show in full.
        
    23. 

  23. 

    
            [root@qas-k8s-master01 ssl]#
        




将可执行文件路/k8s/kubernetes/ 添加到 PATH 变量中




  1. 

    
            vim /etc/profile
        
    2. 

  2. 

    
            PATH=/k8s/kubernetes/bin:$PATH:$HOME/bin
        
    3. 

  3. 

    
            source /etc/profile
        




查看master集群状态




  1. 

    
            [root@qas-k8s-master01 ssl]# kubectl get cs,nodes
        
    2. 

  2. 

    
            NAME STATUS MESSAGE ERROR
        
    3. 

  3. 

    
            componentstatus/controller-manager Healthy ok
        
    4. 

  4. 

    
            componentstatus/scheduler Healthy ok
        
    5. 

  5. 

    
            componentstatus/etcd-2 Healthy {"health":"true"}
        
    6. 

  6. 

    
            componentstatus/etcd-1 Healthy {"health":"true"}
        
    7. 

  7. 

    
            componentstatus/etcd-0 Healthy {"health":"true"}
        
    8. 

  8. 

    9. 

  9. 

    
            [root@qas-k8s-master01 ssl]#
        




5、部署node 节点


kubernetes work 节点运行如下组件：


  • docker 前面已经部署
    • 

  • kubelet
    • 

  • kube-proxy


部署 kubelet 组件


  • kublet 运行在每个 worker 节点上，接收 kube-apiserver 发送的请求，管理 Pod 容器，执行交互式命令，如exec、run、logs 等;
    • 

  • kublet 启动时自动向 kube-apiserver 注册节点信息，内置的 cadvisor 统计和监控节点的资源使用情况;
    • 

  • 为确保安全，本文档只开启接收 https 请求的安全端口，对请求进行认证和授权，拒绝未授权的访问(如apiserver、heapster)。


将kubelet 二进制文件拷贝node节点




  1. 

    
            cp kubelet kube-proxy /k8s/kubernetes/bin/
        
    2. 

  2. 

    
            scp kubelet kube-proxy 192.168.1.165:/k8s/kubernetes/bin/
        
    3. 

  3. 

    
            scp kubelet kube-proxy 192.168.1.215:/k8s/kubernetes/bin/
        




创建 kubelet bootstrap kubeconfig 文件


此脚本在/k8s/kubernetes/ssl下创建




  1. 

    
            vim environment.sh
        
    2. 

  2. 

    
            # 创建kubelet bootstrapping kubeconfig
        
    3. 

  3. 

    
            BOOTSTRAP_TOKEN=2366a641f656a0a025abb4aabda4511b
        
    4. 

  4. 

    
            KUBE_APISERVER="https://192.168.1.108:6443"
        
    5. 

  5. 

    
            # 设置集群参数
        
    6. 

  6. 

    
            kubectl config set-cluster kubernetes \
        
    7. 

  7. 

    
            --certificate-authority=./ca.pem \
        
    8. 

  8. 

    
            --embed-certs=true \
        
    9. 

  9. 

    
            --server=${KUBE_APISERVER} \
        
    10. 

  10. 

    
            --kubeconfig=bootstrap.kubeconfig
        
    11. 

  11. 

    12. 

  12. 

    
            # 设置客户端认证参数
        
    13. 

  13. 

    
            kubectl config set-credentials kubelet-bootstrap \
        
    14. 

  14. 

    
            --token=${BOOTSTRAP_TOKEN} \
        
    15. 

  15. 

    
            --kubeconfig=bootstrap.kubeconfig
        
    16. 

  16. 

    17. 

  17. 

    
            # 设置上下文参数
        
    18. 

  18. 

    
            kubectl config set-context default \
        
    19. 

  19. 

    
            --cluster=kubernetes \
        
    20. 

  20. 

    
            --user=kubelet-bootstrap \
        
    21. 

  21. 

    
            --kubeconfig=bootstrap.kubeconfig
        
    22. 

  22. 

    23. 

  23. 

    
            # 设置默认上下文
        
    24. 

  24. 

    
            kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
        
    25. 

  25. 

    26. 

  26. 

    
            #----------------------
        
    27. 

  27. 

    28. 

  28. 

    
            # 创建kube-proxy kubeconfig文件
        
    29. 

  29. 

    30. 

  30. 

    
            kubectl config set-cluster kubernetes \
        
    31. 

  31. 

    
            --certificate-authority=./ca.pem \
        
    32. 

  32. 

    
            --embed-certs=true \
        
    33. 

  33. 

    
            --server=${KUBE_APISERVER} \
        
    34. 

  34. 

    
            --kubeconfig=kube-proxy.kubeconfig
        
    35. 

  35. 

    36. 

  36. 

    
            kubectl config set-credentials kube-proxy \
        
    37. 

  37. 

    
            --client-certificate=./kube-proxy.pem \
        
    38. 

  38. 

    
            --client-key=./kube-proxy-key.pem \
        
    39. 

  39. 

    
            --embed-certs=true \
        
    40. 

  40. 

    
            --kubeconfig=kube-proxy.kubeconfig
        
    41. 

  41. 

    42. 

  42. 

    
            kubectl config set-context default \
        
    43. 

  43. 

    
            --cluster=kubernetes \
        
    44. 

  44. 

    
            --user=kube-proxy \
        
    45. 

  45. 

    
            --kubeconfig=kube-proxy.kubeconfig
        
    46. 

  46. 

    47. 

  47. 

    
            kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
        




运行脚本




  1. 

    
            [root@qas-k8s-master01 ssl]# pwd
        
    2. 

  2. 

    
            /k8s/kubernetes/ssl
        
    3. 

  3. 

    
            [root@qas-k8s-master01 ssl]# ./environment.sh
        




将bootstrap kubeconfig kube-proxy.kubeconfig 文件拷贝到所有 nodes节点




  1. 

    
            cp bootstrap.kubeconfig kube-proxy.kubeconfig /k8s/kubernetes/cfg/
        
    2. 

  2. 

    
            scp bootstrap.kubeconfig kube-proxy.kubeconfig 192.168.1.165:/k8s/kubernetes/cfg/
        
    3. 

  3. 

    
            scp bootstrap.kubeconfig kube-proxy.kubeconfig 192.168.1.215:/k8s/kubernetes/cfg/
        




创建kubelet 参数配置文件拷贝到所有 nodes节点


创建 kubelet 参数配置模板文件：




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/kubernetes/cfg/kubelet.config
        
    2. 

  2. 

    
            kind: KubeletConfiguration
        
    3. 

  3. 

    
            apiVersion: kubelet.config.k8s.io/v1beta1
        
    4. 

  4. 

    
            address: 192.168.1.108
        
    5. 

  5. 

    
            port: 10250
        
    6. 

  6. 

    
            readOnlyPort: 10255
        
    7. 

  7. 

    
            cgroupDriver: cgroupfs
        
    8. 

  8. 

    
            clusterDNS: ["10.0.0.2"]
        
    9. 

  9. 

    
            clusterDomain: cluster.local.
        
    10. 

  10. 

    
            failSwapOn: false
        
    11. 

  11. 

    
            authentication:
        
    12. 

  12. 

    
            anonymous:
        
    13. 

  13. 

    
            enabled: true
        
    14. 

  14. 

    
            [root@qas-k8s-master01 ssl]#
        




创建kubelet配置文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/kubernetes/cfg/kubelet
        
    2. 

  2. 

    
            KUBELET_OPTS="--logtostderr=true \
        
    3. 

  3. 

    
            --v=4 \
        
    4. 

  4. 

    
            --hostname-override=192.168.1.108 \
        
    5. 

  5. 

    
            --kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig \
        
    6. 

  6. 

    
            --bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig \
        
    7. 

  7. 

    
            --config=/k8s/kubernetes/cfg/kubelet.config \
        
    8. 

  8. 

    
            --cert-dir=/k8s/kubernetes/ssl \
        
    9. 

  9. 

    
            --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
        
    10. 

  10. 

    
            [root@qas-k8s-master01 ssl]#
        




创建kubelet systemd unit 文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/kubelet.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Kubernetes Kubelet
        
    4. 

  4. 

    
            After=docker.service
        
    5. 

  5. 

    
            Requires=docker.service
        
    6. 

  6. 

    7. 

  7. 

    
            [Service]
        
    8. 

  8. 

    
            EnvironmentFile=/k8s/kubernetes/cfg/kubelet
        
    9. 

  9. 

    
            ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS
        
    10. 

  10. 

    
            Restart=on-failure
        
    11. 

  11. 

    
            KillMode=process
        
    12. 

  12. 

    13. 

  13. 

    
            [Install]
        
    14. 

  14. 

    
            WantedBy=multi-user.target
        
    15. 

  15. 

    
            [root@qas-k8s-master01 ssl]#
        




将kubelet-bootstrap用户绑定到系统集群角色




  1. 

    
            kubectl create clusterrolebinding kubelet-bootstrap \
        
    2. 

  2. 

    
            --clusterrole=system:node-bootstrapper \
        
    3. 

  3. 

    
            --user=kubelet-bootstrap
        




启动服务




  1. 

    
            systemctl daemon-reload
        
    2. 

  2. 

    
            systemctl enable kubelet
        
    3. 

  3. 

    
            systemctl restart kubelet
        




approve kubelet CSR 请求


可以手动或自动 approve CSR 请求。推荐使用自动的方式，因为从 v1.8 版本开始，可以自动轮转approve csr 后生成的证书。

    手动 approve CSR 请求

    查看 CSR 列表：




  1. 

    
            # kubectl get csr
        
    2. 

  2. 

    
            NAME AGE REQUESTOR CONDITION
        
    3. 

  3. 

    
            node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs 39m kubelet-bootstrap Pending
        
    4. 

  4. 

    
            node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s 5m5s kubelet-bootstrap Pending
        
    5. 

  5. 

    6. 

  6. 

    
            # kubectl certificate approve node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs
        
    7. 

  7. 

    
            certificatesigningrequest.certificates.k8s.io/node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs
        
    8. 

  8. 

    9. 

  9. 

    
            # kubectl certificate approve node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s
        
    10. 

  10. 

    
            certificatesigningrequest.certificates.k8s.io/node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s approved
        
    11. 

  11. 

    12. 

  12. 

    
            # kubectl get csr
        
    13. 

  13. 

    
            NAME AGE REQUESTOR CONDITION
        
    14. 

  14. 

    
            node-csr-An1VRgJ7FEMMF_uyy6iPjyF5ahuLx6tJMbk2SMthwLs 41m kubelet-bootstrap Approved,Issued
        
    15. 

  15. 

    
            node-csr-dWPIyP_vD1w5gBS4iTZ6V5SJwbrdMx05YyybmbW3U5s 7m32s kubelet-bootstrap Approved,Issued
        




  • Requesting User：请求 CSR 的用户，kube-apiserver 对它进行认证和授权；
    • 

  • Subject：请求签名的证书信息；
    • 

  • 证书的 CN 是 system:node:kube-node2， Organization 是 system:nodes，kube-apiserver 的 Node 授权模式会授予该证书的相关权限；


查看集群状态




  1. 

    
            [root@qas-k8s-master01 bin]# kubectl get nodes
        
    2. 

  2. 

    
            NAME STATUS ROLES AGE VERSION
        
    3. 

  3. 

    
            192.168.1.108 Ready master 68m v1.15.0
        
    4. 

  4. 

    
            192.168.1.165 Ready node 68m v1.15.0
        
    5. 

  5. 

    
            192.168.1.215 Ready node 68m v1.15.0
        
    6. 

  6. 

    
            [root@qas-k8s-master01 bin]#
        




部署 kube-proxy 组件


kube-proxy 运行在所有 node节点上，它监听 apiserver 中 service 和 Endpoint 的变化情况，创建路由规则来进行服务负载均衡。


创建 kube-proxy 配置文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /k8s/kubernetes/cfg/kube-proxy
        
    2. 

  2. 

    
            KUBE_PROXY_OPTS="--logtostderr=true \
        
    3. 

  3. 

    
            --v=4 \
        
    4. 

  4. 

    
            --hostname-override=192.168.1.108 \
        
    5. 

  5. 

    
            --cluster-cidr=10.0.0.0/24 \
        
    6. 

  6. 

    
            --kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubeconfig"
        
    7. 

  7. 

    
            [root@qas-k8s-master01 ssl]#
        




  • bindAddress: 监听地址；
    • 

  • clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件；
    • 

  • clusterCIDR: kube-proxy 根据 –cluster-cidr 判断集群内部和外部流量，指定 –cluster-cidr 或 –masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT；
    • 

  • hostnameOverride: 参数值必须与 kubelet 的值一致，否则 kube-proxy 启动后会找不到该 Node，从而不会创建任何 ipvs 规则；
    • 

  • mode: 使用 ipvs 模式；


创建kube-proxy systemd unit 文件




  1. 

    
            [root@qas-k8s-master01 ssl]# cat /usr/lib/systemd/system/kube-proxy.service
        
    2. 

  2. 

    
            [Unit]
        
    3. 

  3. 

    
            Description=Kubernetes Proxy
        
    4. 

  4. 

    
            After=network.target
        
    5. 

  5. 

    6. 

  6. 

    
            [Service]
        
    7. 

  7. 

    
            EnvironmentFile=-/k8s/kubernetes/cfg/kube-proxy
        
    8. 

  8. 

    
            ExecStart=/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
        
    9. 

  9. 

    
            Restart=on-failure
        
    10. 

  10. 

    11. 

  11. 

    
            [Install]
        
    12. 

  12. 

    
            WantedBy=multi-user.target
        
    13. 

  13. 

    
            [root@qas-k8s-master01 ssl]#
        




启动服务




  1. 

    
            systemctl daemon-reload
        
    2. 

  2. 

    
            systemctl enable kube-proxy
        
    3. 

  3. 

    
            systemctl restart kube-proxy
        




查看 kube-proxy状态




  1. 

    
            [root@qas-k8s-master01 ssl]# systemctl status kube-proxy
        
    2. 

  2. 

    
            ● kube-proxy.service - Kubernetes Proxy
        
    3. 

  3. 

    
            Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
        
    4. 

  4. 

    
            Active: active (running) since Tue 2019-07-09 15:56:51 CST; 55min ago
        
    5. 

  5. 

    
            Main PID: 30062 (kube-proxy)
        
    6. 

  6. 

    
            Tasks: 0
        
    7. 

  7. 

    
            Memory: 43.0M
        
    8. 

  8. 

    
            CGroup: /system.slice/kube-proxy.service
        
    9. 

  9. 

    
            ‣ 30062 /k8s/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=192.168.1.108 --cluster-cidr=10.0....
        
    10. 

  10. 

    11. 

  11. 

    
            Jul 09 16:51:58 qas-k8s-master01 kube-proxy[30062]: I0709 16:51:58.710766 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    12. 

  12. 

    
            Jul 09 16:51:59 qas-k8s-master01 kube-proxy[30062]: I0709 16:51:59.594996 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    13. 

  13. 

    
            Jul 09 16:52:00 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:00.717813 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    14. 

  14. 

    
            Jul 09 16:52:01 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:01.608540 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    15. 

  15. 

    
            Jul 09 16:52:02 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:02.734118 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    16. 

  16. 

    
            Jul 09 16:52:03 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:03.629360 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    17. 

  17. 

    
            Jul 09 16:52:04 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:04.742294 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    18. 

  18. 

    
            Jul 09 16:52:05 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:05.636544 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    19. 

  19. 

    
            Jul 09 16:52:06 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:06.752374 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    20. 

  20. 

    
            Jul 09 16:52:07 qas-k8s-master01 kube-proxy[30062]: I0709 16:52:07.646475 30062 config.go:132] Calling handler.OnEndpointsUpdate
        
    21. 

  21. 

    
            [root@qas-k8s-master01 ssl]#
        




集群状态


打node 或者master 节点的标签




  1. 

    
            kubectl label node 192.168.1.108 node-role.kubernetes.io/master='master'
        
    2. 

  2. 

    
            kubectl label node 192.168.1.165 node-role.kubernetes.io/node='node'
        
    3. 

  3. 

    
            kubectl label node 192.168.1.215 node-role.kubernetes.io/node='node'
        






  1. 

    
            [root@qas-k8s-master01 bin]# kubectl get node,cs
        
    2. 

  2. 

    
            NAME STATUS ROLES AGE VERSION
        
    3. 

  3. 

    
            node/192.168.1.108 Ready master 69m v1.15.0
        
    4. 

  4. 

    
            node/192.168.1.165 Ready node 70m v1.15.0
        
    5. 

  5. 

    
            node/192.168.1.215 Ready node 69m v1.15.0
        
    6. 

  6. 

    7. 

  7. 

    
            NAME STATUS MESSAGE ERROR
        
    8. 

  8. 

    
            componentstatus/scheduler Healthy ok
        
    9. 

  9. 

    
            componentstatus/controller-manager Healthy ok
        
    10. 

  10. 

    
            componentstatus/etcd-0 Healthy {"health":"true"}
        
    11. 

  11. 

    
            componentstatus/etcd-2 Healthy {"health":"true"}
        
    12. 

  12. 

    
            componentstatus/etcd-1 Healthy {"health":"true"}
        
    13. 

  13. 

    
            [root@qas-k8s-master01 bin]#
        


												


											

CentOS7 使用二进制部署 Kubernetes 1.15-1.17集群（均通用，已经尝试，细心）相关推荐
	

    
						
  1. CentOS7 使用二进制部署 Kubernetes v1.15.3集群
		
    组件版本 && 集群环境 组件版本: Kubernetes v1.15.3 Etcd v3.3.10 Flanneld v0.11.0 服务器IP 角色 192.168.1.241 m ...
    
		
    2. 
								
  2. 二进制部署 单Master Kubernetes-v1.14.1集群
		
    一.部署Kubernetes集群 1.1 Kubernetes介绍 Kubernetes(K8S)是Google开源的容器集群管理系统,K8S在Docker容器技术的基础之上,大大地提高了容器化部署应 ...
    
		
    3. 
						
  3. 二进制部署Kubernetes v1.13.4 HA可选
		
    本次采用二进制文件方式部署,本文过程写成了更详细的ansible部署方案 https://github.com/zhangguanzhang/Kubernetes-ansible 和之前的步骤差不多都 ...
    
		
    4. 
						
  4. CentOS 使用二进制部署 Kubernetes 1.13集群
		
    CentOS 使用二进制部署 Kubernetes 1.13集群 一.概述 kubernetes 1.13 已发布,这是 2018 年年内第四次也是最后一次发布新版本.Kubernetes 1.13  ...
    
		
    5. 
						
  5. 使用ansible部署CDH 5.15.1大数据集群
		
    使用ansible离线部署CDH 5.15.1大数据集群 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 在此之前,我之前分享过使用shell自定义脚本部署大数据集群,不管是部署CD ...
    
		
    6. 
						
  6. kubernetes V1.6.4 分布式集群的部署及service负载均衡
		
    1,kubernetes的基本概念和术语 1.1 kubernetes与微服务的简介 最近几年,微服务一词经常被IT的技术界人士提及,简单介绍的话,微服务架构就是将原本单独部署运行的大型软件拆分为一个 ...
    
		
    7. 
						
  7. Kubernetes(k8s)集群部署（k8s企业级Docker容器集群管理）系列目录
		
    0.目录 整体架构目录:ASP.NET Core分布式项目实战-目录 k8s架构目录:Kubernetes(k8s)集群部署(k8s企业级Docker容器集群管理)系列目录 一.感谢 在此感谢.net ...
    
		
    8. 
						
  8. 【云原生之kubernetes实战】在k8s集群下部署Weave Scope监控平台
		
    [云原生之kubernetes实战]在k8s集群下部署Weave Scope监控平台 一.Weave Scope介绍 1.Weave Scope简介 2.Weave Scope的特点 3.Weave  ...
    
		
    9. 
						
  9. kubernetes使用ansible快速构建集群
		
    软硬件限制: 1)cpu和内存 master:至少1c2g,推荐2c4g:node:至少1c2g 2)linux系统 内核版本至少3.10,推荐CentOS7/RHEL7 3)docker 至少1.9 ...
    
		
    10. 
		


					

最新文章
	

    
						
  1. Kotlin 使用list.add 时候报错的处理方法
		
    2. 
						
  2. JXJJOI2018_T1_market
		
    3. 
						
  3. 不用写代码就能做高端科学计算，Mathematica推出“人话”版软件
		
    4. 
						
  4. Spring Boot学习笔记（1）
		
    5. 
						
  5. TCP协议-如何保证传输可靠性
		
    6. 
						
  6. 《Java8实战》笔记（13）：函数式的思考
		
    7. 
						
  7. linux查看删除init内容,linux常用命令
		
    8. 
						
  8. GNS3（eNSP）和VMWare搭建网络学习环境
		
    9. 
						
  9. 树莓派linux系统识别u盘启动,使用U-Boot让树莓派从U盘启动
		
    10. 
						
  10. mysql集群异地部署_linux 环境下 部署mysql 集群
		
    11. 
						
  11. IDC：阿里云安全能力和IaaS市场份额双项领先
		
    12. 
						
  12. 【资料目录收藏】.NET开发必看资料53个 经典源码77个
		
    13. 
						
  13. c/c++ 去掉空格函数
		
    14. 
						
  14. Trend趋势反垃圾邮件黑名单申诉方法
		
    15. 
						
  15. java拨号上网,Android实现pppoe拨号上网（二）具体步骤
		
    16. 
						
  16. veket linux能运行qq么,【veket系统】Veket Linux系统下载 v8.07 官方正式版-开心电玩...
		
    17. 
						
  17. 为设计指定输入端口驱动强度：set_driving_cell、set_drive 和set_input_transition
		
    18. 
						
  18. Python基础02-蟒蛇绘制
		
    19. 
						
  19. Kibana：如何在 Dashboard 中针对部分的数据做可视化
		
    20. 
						
  20. 【软件工程】--设计阶段
		
    21. 
		

	


热门文章
	

    
									
  1. 用友NC创建表空间、用户、授权
			
    2. 
						
  2. 【Canvas】js如何设置canvas绕图形中心旋转
			
    3. 
						
  3. 1 ，storm 框架介绍
			
    4. 
						
  4. 使用windows钩子捕获进程的启动和关闭消息
			
    5. 
						
  5. Dubbo从入门到实战
			
    6. 
						
  6. Datadome 最新逆向分析(支持爱马仕)
			
    7. 
						
  7. python常用的案例
			
    8. 
						
  8. 《解析深度学习》部分笔记
			
    9. 
						
  9. 智慧家庭信息安全白皮书发布，推动智慧家庭叫好又叫座
			
    10. 
						
  10. pip报错：No module named pip
			
    11.