【谷粒商城】集群篇-k8s(4/4)
2024-05-14 23:27:48
【谷粒商城】集群篇-k8s(4/4)
目录
一、K8s快速入门
1）简介
2）架构
3）概念
4）快速体验
二、K8s集群安装
1）kubeadm
2）前置要求
3）部署步骤
4）环境准备
5）所有节点安装docker、kubeadm、kubelet、kubectl
6）部署k8s-master
7）安装POD网络插件（CNI）
8）加入kubenetes的Node节点
9）入门操作kubernetes集群
三、docker深入
四、K8s细节
1、kubectl文档
2、资源类型
3、格式化输出
命令参考
service的意义
Ingress
安装kubernetes可视化界面——DashBoard
kubesphere
1、简洁
2、安装前提提交
版权
笔记-基础篇-1(P1-P28)：https://blog.csdn.net/hancoder/article/details/106922139
笔记-基础篇-2(P28-P100)：https://blog.csdn.net/hancoder/article/details/107612619
笔记-高级篇(P340)：https://blog.csdn.net/hancoder/article/details/107612746
笔记-vue：https://blog.csdn.net/hancoder/article/details/107007605
笔记-elastic search、上架、检索：https://blog.csdn.net/hancoder/article/details/113922398
笔记-认证服务：https://blog.csdn.net/hancoder/article/details/114242184
笔记-分布式锁与缓存：https://blog.csdn.net/hancoder/article/details/114004280
笔记-集群篇：https://blog.csdn.net/hancoder/article/details/107612802
springcloud笔记：https://blog.csdn.net/hancoder/article/details/109063671
笔记版本说明：2020年提供过笔记文档，但只有P1-P50的内容，2021年整理了P340的内容。请点击标题下面分栏查看系列笔记
声明：
- 可以白嫖，但请勿转载发布，笔记手打不易
- 本系列笔记不断迭代优化，csdn：hancoder上是最新版内容，10W字都是在csdn免费开放观看的。
- 离线md笔记文件获取方式见文末。2021-3版本的md笔记打完压缩包共500k（云图床），包括本项目笔记，还有cloud、docker、mybatis-plus、rabbitMQ等个人相关笔记
sql：https://github.com/FermHan/gulimall/sql文件
本项目其他笔记见专栏：https://blog.csdn.net/hancoder/category_10822407.html
一、K8s快速入门
1）简介
时光回溯
让我们回顾一下为什么 Kubernetes 如此有用。
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-jhiwhOpn-1620146330110)(https://d33wubrfki0l68.cloudfront.net/26a177ede4d7b032362289c6fccd448fc4a91174/eb693/images/docs/container_evolution.svg)]
传统部署时代：
早期，各个组织机构在物理服务器上运行应用程序。无法为物理服务器中的应用程序定义资源边界，这会导致资源分配问题。 例如，如果在物理服务器上运行多个应用程序，则可能会出现一个应用程序占用大部分资源的情况， 结果可能导致其他应用程序的性能下降。 一种解决方案是在不同的物理服务器上运行每个应用程序，但是由于资源利用不足而无法扩展， 并且维护许多物理服务器的成本很高。
虚拟化部署时代：
作为解决方案，引入了虚拟化。虚拟化技术允许你在单个物理服务器的 CPU 上运行多个虚拟机（VM）。 虚拟化允许应用程序在 VM 之间隔离，并提供一定程度的安全，因为一个应用程序的信息 不能被另一应用程序随意访问。
虚拟化技术能够更好地利用物理服务器上的资源，并且因为可轻松地添加或更新应用程序 而可以实现更好的可伸缩性，降低硬件成本等等。
每个 VM 是一台完整的计算机，在虚拟化硬件之上运行所有组件，包括其自己的操作系统。
容器部署时代：
容器类似于 VM，但是它们具有被放宽的隔离属性，可以在应用程序之间共享操作系统（OS）。 因此，容器被认为是轻量级的。容器与 VM 类似，具有自己的文件系统、CPU、内存、进程空间等。 由于它们与基础架构分离，因此可以跨云和 OS 发行版本进行移植。
容器因具有许多优势而变得流行起来。下面列出的是容器的一些好处：
- 敏捷应用程序的创建和部署：与使用 VM 镜像相比，提高了容器镜像创建的简便性和效率。
- 持续开发、集成和部署：通过快速简单的回滚（由于镜像不可变性），支持可靠且频繁的 容器镜像构建和部署。
- 关注开发与运维的分离：在构建/发布时而不是在部署时创建应用程序容器镜像， 从而将应用程序与基础架构分离。
- 可观察性不仅可以显示操作系统级别的信息和指标，还可以显示应用程序的运行状况和其他指标信号。
- 跨开发、测试和生产的环境一致性：在便携式计算机上与在云中相同地运行。
- 跨云和操作系统发行版本的可移植性：可在 Ubuntu、RHEL、CoreOS、本地、 Google Kubernetes Engine 和其他任何地方运行。
- 以应用程序为中心的管理：提高抽象级别，从在虚拟硬件上运行 OS 到使用逻辑资源在 OS 上运行应用程序。
- 松散耦合、分布式、弹性、解放的微服务：应用程序被分解成较小的独立部分， 并且可以动态部署和管理 - 而不是在一台大型单机上整体运行。
- 资源隔离：可预测的应用程序性能。
- 资源利用：高效率和高密度。
2）架构
（1）整体主从方式
（2）master节点架构
（3）Node节点架构
3）概念
4）快速体验
（1）安装minikube
https://github.com/kubernetes/minikube/releases
下载minikuber-windows-amd64.exe 改名为minikube.exe
打开virtualBox，打开cmd
运行
minikube start --vm-driver=virtualbox --registry-mirror=https://registry.docker-cn.com
等待20分钟即可。
（2）体验nginx部署升级
提交一个nginx deployment
kubectl apply -f https://k8s.io/examples/application/deployment.yaml升级 nginx deployment
kubectl apply -f https://k8s.io/examples/application/deployment-update.yaml扩容 nginx deployment
二、K8s集群安装
1）kubeadm
kubeadm是官方社区推出的一个用于快速部署kuberneters集群的工具。
这个工具能通过两条指令完成一个kuberneters集群的部署
$ kuberneters init
$ kubeadm join <Master节点的IP和端口>
2）前置要求
3）部署步骤
- 在所有的节点上安装Docker和kubeadm
- 不是Kubernetes Master
- 部署容器网络插件
- 部署Kubernetes Node，将节点加入Kubernetes集群中
- 部署DashBoard web页面，可视化查看Kubernetes资源
4）环境准备
（1）准备工作
- 我们可以使用vagrant快速创建三个虚拟机。虚拟机启动前先设置virtualbox的主机网络。现在全部统一为192.168.56.1，以后所有虚拟机都是56.x的ip地址。
- 在全局设定中，找到一个空间比较大的磁盘用用来存放镜像。
网卡1是NAT，用于虚拟机与本机访问互联网。网卡2是仅主机网络，虚拟机内部共享的虚拟网络
（2）启动三个虚拟机
- 使用我们提供的vagrant文件，复制到非中文无空格目录下，运行vagrant up启动三个虚拟机。其实vagrant完全可以一键部署全部K8s集群
https://github.com/rootsongjc/kubernetes-vagrant-centos-cluster
http://github.com/davidkbainbridge/k8s-playground
下面是vagrantfile，使用它来创建三个虚拟机，分别为k8s-node1，k8s-node2和k8s-node3.
Vagrant.configure("2") do |config|(1..3).each do |i|config.vm.define "k8s-node#{i}" do |node|# 设置虚拟机的Boxnode.vm.box = "centos/7"# 设置虚拟机的主机名node.vm.hostname="k8s-node#{i}"# 设置虚拟机的IPnode.vm.network "private_network", ip: "192.168.56.#{99+i}", netmask: "255.255.255.0"# 设置主机与虚拟机的共享目录# node.vm.synced_folder "~/Documents/vagrant/share", "/home/vagrant/share"# VirtaulBox相关配置node.vm.provider "virtualbox" do |v|# 设置虚拟机的名称v.name = "k8s-node#{i}"# 设置虚拟机的内存大小v.memory = 4096# 设置虚拟机的CPU个数v.cpus = 4endendend
end
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 进入到三个虚拟机，开启root的密码访问权限
vagrant ssh xxx进入到系统后
# vagrant ssh k8s-node1
su root 密码为vagrantvi /etc/ssh/sshd_config修改
PermitRootLogin yes
PasswordAuthentication yes所有的虚拟机设为4核4G
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
service sshd restart
- 1
192.168.56.100:22
- 1
关于在"网络地址转换"的连接方式下，三个节点的eth0，IP地址相同的问题。
**问题描述：**查看k8s-node1的路由表：
# 查看默认的网卡
[root@k8s-node1 ~]# ip route show
default via 10.0.2.2 dev eth0 proto dhcp metric 100 10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100 192.168.56.0/24 dev eth1 proto kernel scope link src 192.168.56.100 metric 101
- 1
- 2
- 3
- 4
- 5
- 6
- 7
能够看到路由表中记录的是，通过端口eth0进行数据包的收发。
分别查看k8s-node1，k8s-node2和k8s-node3的eth0所绑定的IP地址，发现它们都是相同的，全都是10.0.2.15，这些地址是供kubernetes集群通信用的，区别于eth1上的IP地址，是通远程管理使用的。
[root@k8s-node1 ~]# ip addr
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 52:54:00:8a:fe:e6 brd ff:ff:ff:ff:ff:ffinet 10.0.2.15/24 brd 10.0.2.255 scope global noprefixroute dynamic eth0valid_lft 84418sec preferred_lft 84418secinet6 fe80::5054:ff:fe8a:fee6/64 scope link valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 08:00:27:a3:ca:c0 brd ff:ff:ff:ff:ff:ffinet 192.168.56.100/24 brd 192.168.56.255 scope global noprefixroute eth1valid_lft forever preferred_lft foreverinet6 fe80::a00:27ff:fea3:cac0/64 scope link valid_lft forever preferred_lft forever
[root@k8s-node1 ~]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
**原因分析：**这是因为它们使用是端口转发规则，使用同一个地址，通过不同的端口来区分。但是这种端口转发规则在以后的使用中会产生很多不必要的问题，所以需要修改为NAT网络类型。
解决方法：
- 选择三个节点，然后执行“管理”->“全局设定”->“网络”，添加一个NAT网络。
- 分别修改每台设备的网络类型，并刷新重新生成MAC地址。
刷新一下MAC地址
1网络是集群交互，2网络是宿主交互
- 再次查看三个节点的IP
（3）设置Linux环境（三个节点都执行）
- 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
- 1
- 2
- 关闭seLinux
# linux默认的安全策略
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0
- 1
- 2
- 3
- 关闭swap
swapoff -a #临时关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab #永久关闭
free -g #验证，swap必须为0
- 1
- 2
- 3
- 添加主机名与IP对应关系：
查看主机名：
hostname
- 1
如果主机名不正确，可以通过“hostnamectl set-hostname <newhostname> :指定新的hostname”命令来进行修改。
vi /etc/hosts
10.0.2.4 k8s-node1
10.0.2.5 k8s-node2
10.0.2.6 k8s-node3
- 1
- 2
- 3
- 4
将桥接的IPV4流量传递到iptables的链：
cat > /etc/sysctl.d/k8s.conf <<EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOF
- 1
- 2
- 3
- 4
- 5
- 6
- 7
应用规则：
sysctl --system
- 1
疑难问题：遇见提示是只读的文件系统，运行如下命令
mount -o remount rw /
- 1
- date 查看时间（可选）
yum -y install ntpupdatentpupdate time.window.com #同步最新时间
- 1
- 2
- 3
5）所有节点安装docker、kubeadm、kubelet、kubectl
Kubenetes默认CRI（容器运行时）为Docker，因此先安装Docker。
（1）安装Docker
1、卸载之前的docker
sudo yum remove docker \docker-client \docker-client-latest \docker-common \docker-latest \docker-latest-logrotate \docker-logrotate \docker-engine
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
2、安装Docker -CE
sudo yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
# 设置docker repo的yum位置
sudo yum-config-manager \--add-repo \https://download.docker.com/linux/centos/docker-ce.repo# 安装docker，docker-cli
sudo yum -y install docker-ce docker-ce-cli containerd.io
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
3、配置docker加速
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{"registry-mirrors": ["https://ke9h1pt4.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
4、启动Docker && 设置docker开机启动
systemctl enable docker
- 1
基础环境准备好，可以给三个虚拟机备份一下；
（2）添加阿里与Yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
更多详情见： https://developer.aliyun.com/mirror/kubernetes
（3）安装kubeadm，kubelet和kubectl
yum list|grep kube
- 1
安装
yum install -y kubelet-1.17.3 kubeadm-1.17.3 kubectl-1.17.3
- 1
开机启动
systemctl enable kubelet && systemctl start kubelet
- 1
查看kubelet的状态：
systemctl status kubelet
- 1
查看kubelet版本：
[root@k8s-node2 ~]# kubelet --version
Kubernetes v1.17.3
- 1
- 2
6）部署k8s-master
（1）master节点初始化
在Master节点上，创建并执行master_images.sh
#!/bin/bashimages=(kube-apiserver:v1.17.3kube-proxy:v1.17.3kube-controller-manager:v1.17.3kube-scheduler:v1.17.3coredns:1.6.5etcd:3.4.3-0pause:3.1
)for imageName in ${images[@]} ; dodocker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
done
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
查看100的内部通信端口
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 08:00:27:7e:dd:f5 brd ff:ff:ff:ff:ff:ffinet 10.0.2.4/24 brd 10.0.2.255 scope global noprefixroute dynamic eth0valid_lft 512sec preferred_lft 512secinet6 fe80::a00:27ff:fe7e:ddf5/64 scope linkvalid_lft forever preferred_lft forever
- 1
- 2
- 3
- 4
- 5
- 6
初始化kubeadm
kubeadm init \
--apiserver-advertise-address=10.0.2.4 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--kubernetes-version v1.17.3 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=10.244.0.0/16
- 1
- 2
- 3
- 4
- 5
- 6
注：
- –apiserver-advertise-address=10.0.2.21 ：这里的IP地址是master主机的地址，为上面的eth0网卡的地址；
执行结果：
[root@k8s-node1 opt]# kubeadm init \
> --apiserver-advertise-address=10.0.2.15 \
> --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
> --kubernetes-version v1.17.3 \
> --service-cidr=10.96.0.0/16 \
> --pod-network-cidr=10.244.0.0/16
W0503 14:07:12.594252 10124 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.17.3
[preflight] Running pre-flight checks[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.2.15]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-node1 localhost] and IPs [10.0.2.15 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-node1 localhost] and IPs [10.0.2.15 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0503 14:07:30.908642 10124 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0503 14:07:30.911330 10124 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 22.506521 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: sg47f3.4asffoi6ijb8ljhq
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
#表示kubernetes已经初始化成功了
Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 10.0.2.15:6443 --token sg47f3.4asffoi6ijb8ljhq \--discovery-token-ca-cert-hash sha256:81fccdd29970cbc1b7dc7f171ac0234d53825bdf9b05428fc9e6767436991bfb
[root@k8s-node1 opt]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
由于默认拉取镜像地址k8s.cr.io国内无法访问，这里指定阿里云仓库地址。可以手动按照我们的images.sh先拉取镜像。
地址变为：registry.aliyuncs.com/googole_containers也可以。
科普：无类别域间路由（Classless Inter-Domain Routing 、CIDR）是一个用于给用户分配IP地址以及在互联网上有效第路由IP数据包的对IP地址进行归类的方法。
拉取可能失败，需要下载镜像。
运行完成提前复制：加入集群的令牌。
（2）测试Kubectl（主节点执行）
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 1
- 2
- 3
详细部署文档：https://kubernetes.io/docs/concepts/cluster-administration/addons/
kubectl get nodes #获取所有节点
- 1
目前Master状态为notready。等待网络加入完成即可。
journalctl -u kubelet #查看kubelet日志
- 1
kubeadm join 10.0.2.4:6443 --token bt3hkp.yxnpzsgji4a6edy7 \--discovery-token-ca-cert-hash sha256:64949994a89c53e627d68b115125ff753bfe6ff72a26eb561bdc30f32837415a
- 1
- 2
7）安装POD网络插件（CNI）
kubectl apply -f \
https://raw.githubusercontent.com/coreos/flanne/master/Documentation/kube-flannel.yml
[root@k8s-node1 k8s]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
[root@k8s-node1 k8s]#
[root@k8s-node1 k8s]# kubectl get ns
NAME STATUS AGE
default Active 30m
kube-node-lease Active 30m
kube-public Active 30m
kube-system Active 30m
[root@k8s-node1 k8s]#
[root@k8s-node1 k8s]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-546565776c-9sbmk 0/1 Pending 0 31m
kube-system coredns-546565776c-t68mr 0/1 Pending 0 31m
kube-system etcd-k8s-node1 1/1 Running 0 31m
kube-system kube-apiserver-k8s-node1 1/1 Running 0 31m
kube-system kube-controller-manager-k8s-node1 1/1 Running 0 31m
kube-system kube-flannel-ds-amd64-6xwth 1/1 Running 0 2m50s
kube-system kube-proxy-sz2vz 1/1 Running 0 31m
kube-system kube-scheduler-k8s-node1 1/1 Running 0 31m
[root@k8s-node1 k8s]#
[root@k8s-node1 k8s]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready master 34m v1.17.3 #status为ready才能够执行下面的命令
[root@k8s-node1 k8s]#
最后再次执行，并且分别在“k8s-node2”和“k8s-node3”上也执行这里命令：
kubeadm join 10.0.2.4:6443 --token bt3hkp.yxnpzsgji4a6edy7 \--discovery-token-ca-cert-hash sha256:64949994a89c53e627d68b115125ff753bfe6ff72a26eb561bdc30f32837415a
[root@k8s-node1 opt]# kubectl get nodes;
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready master 47m v1.17.3
k8s-node2 NotReady <none> 75s v1.17.3
k8s-node3 NotReady <none> 76s v1.17.3
[root@k8s-node1 opt]#
# 在master执行
watch kubectl get pod -n kube-system -o wide
等到所有的status都变为running状态后，再次查看节点信息：
[root@k8s-node1 ~]# kubectl get nodes;
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready master 3h50m v1.17.3
k8s-node2 Ready <none> 3h3m v1.17.3
k8s-node3 Ready <none> 3h3m v1.17.3
[root@k8s-node1 ~]#
8）加入kubenetes的Node节点
9）入门操作kubernetes集群
1、在主节点上部署一个tomcat
kubectl create deployment tomcat6 --image=tomcat:6.0.53-jre8
- 1
获取所有的资源：
[root@k8s-node1 k8s]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tomcat6-7b84fb5fdc-cfd8g 0/1 ContainerCreating 0 41sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 70mNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tomcat6 0/1 1 0 41sNAME DESIRED CURRENT READY AGE
replicaset.apps/tomcat6-7b84fb5fdc 1 1 0 41s
[root@k8s-node1 k8s]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
kubectl get pods -o wide 可以获取到tomcat部署信息，能够看到它被部署到了k8s-node3上了
[root@k8s-node1 k8s]# kubectl get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/tomcat6-5f7ccf4cb9-xhrr9 0/1 ContainerCreating 0 77s <none> k8s-node3 <none> <none>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 68m <none>NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/tomcat6 0/1 1 0 77s tomcat tomcat:6.0.53-jre8 app=tomcat6NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/tomcat6-5f7ccf4cb9 1 1 0 77s tomcat tomcat:6.0.53-jre8 app=tomcat6,pod-template-hash=5f7ccf4cb9
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
查看node3节点上，下载了哪些镜像：
[root@k8s-node3 k8s]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy v1.17.3 ae853e93800d 14 months ago 116MB
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 2 years ago 52.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.1 da86e6ba6ca1 3 years ago 742kB
tomcat 6.0.53-jre8 49ab0583115a 3 years ago 290MB- 1
- 2
- 3
- 4
- 5
- 6
- 7
查看Node3节点上，正在运行的容器：
[root@k8s-node3 k8s]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8a197fa41dd9 tomcat "catalina.sh run" About a minute ago Up About a minute k8s_tomcat_tomcat6-5f7ccf4cb9-xhrr9_default_81f186a8-4805-4bbb-8d77-3142269942ed_0
4074d0d63a88 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 "/pause" 2 minutes ago Up 2 minutes k8s_POD_tomcat6-5f7ccf4cb9-xhrr9_default_81f186a8-4805-4bbb-8d77-3142269942ed_0
db3faf3a280d ff281650a721 "/opt/bin/flanneld -…" 29 minutes ago Up 29 minutes k8s_kube-flannel_kube-flannel-ds-amd64-vcktd_kube-system_31ca3556-d6c3-48b2-b393-35ff7d89a078_0
be461b54cb4b registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy "/usr/local/bin/kube…" 30 minutes ago Up 30 minutes k8s_kube-proxy_kube-proxy-ptq2t_kube-system_0e1f7df3-7204-481d-bf15-4b0e09cf0c81_0
88d1ab87f400 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 "/pause" 31 minutes ago Up 31 minutes k8s_POD_kube-flannel-ds-amd64-vcktd_kube-system_31ca3556-d6c3-48b2-b393-35ff7d89a078_0
52be28610a02 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 "/pause" 31 minutes ago Up 31 minutes k8s_POD_kube-proxy-ptq2t_kube-system_0e1f7df3-7204-481d-bf15-4b0e09cf0c81_0
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
在node1上执行：
[root@k8s-node1 k8s]# kubectl get pods
NAME READY STATUS RESTARTS AGE
tomcat6-7b84fb5fdc-cfd8g 1/1 Running 0 5m35s[root@k8s-node1 k8s]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default tomcat6-7b84fb5fdc-cfd8g 1/1 Running 0 163m
kube-system coredns-546565776c-9sbmk 1/1 Running 0 3h52m
kube-system coredns-546565776c-t68mr 1/1 Running 0 3h52m
kube-system etcd-k8s-node1 1/1 Running 0 3h52m
kube-system kube-apiserver-k8s-node1 1/1 Running 0 3h52m
kube-system kube-controller-manager-k8s-node1 1/1 Running 0 3h52m
kube-system kube-flannel-ds-amd64-5xs5j 1/1 Running 0 3h6m
kube-system kube-flannel-ds-amd64-6xwth 1/1 Running 0 3h24m
kube-system kube-flannel-ds-amd64-fvnvx 1/1 Running 0 3h6m
kube-system kube-proxy-7tkvl 1/1 Running 0 3h6m
kube-system kube-proxy-mvlnk 1/1 Running 0 3h6m
kube-system kube-proxy-sz2vz 1/1 Running 0 3h52m
kube-system kube-scheduler-k8s-node1 1/1 Running 0 3h52m
[root@k8s-node1 ~]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
从前面看到tomcat部署在Node3上，现在模拟因为各种原因宕机的情况，将node3关闭电源，观察情况。
docker stop执行的时候，docker ps发现又有新的容器了，这是k8s又新建了，所以选择关机node3
[root@k8s-node1 k8s]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-node1 Ready master 79m v1.17.3
k8s-node2 Ready <none> 41m v1.17.3
k8s-node3 NotReady <none> 41m v1.17.3
- 1
- 2
- 3
- 4
- 5
得等个几分钟才能容灾恢复
[root@k8s-node1 k8s]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
tomcat6-5f7ccf4cb9-clcpr 1/1 Running 0 4m16s 10.244.1.2 k8s-node2 <none> <none>
tomcat6-5f7ccf4cb9-xhrr9 1/1 Terminating 1 22m 10.244.2.2 k8s-node3 <none> <none>
- 1
- 2
- 3
- 4
2、暴露nginx访问
在master上执行
kubectl expose deployment tomcat6 --port=80 --target-port=8080 --type=NodePort
- 1
pod的80映射容器的8080；server会带来pod的80
查看服务：
[root@k8s-node1 k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 93m
tomcat6 NodePort 10.96.7.78 <none> 80:30055/TCP 8s
- 1
- 2
- 3
- 4
[root@k8s-node1 k8s]# kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 103m <none>
tomcat6 NodePort 10.96.7.78 <none> 80:30055/TCP 9m38s app=tomcat6
- 1
- 2
- 3
- 4
浏览器输入：http://192.168.56.100:30055/ ，可以看到tomcat首页
输入下面命令可以看到pod和封装pod 的service，pod是部署产生的，部署还有一个副本
[root@k8s-node1 ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tomcat6-5f7ccf4cb9-clcpr 1/1 Running 0 4h12mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h37m
service/tomcat6 NodePort 10.96.7.78 <none> 80:30055/TCP 4h3mNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tomcat6 1/1 1 1 4h30mNAME DESIRED CURRENT READY AGE
replicaset.apps/tomcat6-5f7ccf4cb9 1 1 1 4h30m- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
3、动态扩容测试
kubectl get deployment
[root@k8s-node1 ~]# kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
tomcat6 2/2 2 2 11h
[root@k8s-node1 ~]#
- 1
- 2
- 3
- 4
应用升级： kubectl set image (–help查看帮助)
扩容：kubectl scale --replicas=3 deployment tomcat6
[root@k8s-node1 ~]# kubectl scale --replicas=3 deployment tomcat6
deployment.apps/tomcat6 scaled
[root@k8s-node1 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
tomcat6-5f7ccf4cb9-clcpr 1/1 Running 0 4h23m 10.244.1.2 k8s-node2 <none> <none>
tomcat6-5f7ccf4cb9-jbvr4 1/1 Running 0 9s 10.244.2.3 k8s-node3 <none> <none>
tomcat6-5f7ccf4cb9-ng556 1/1 Running 0 9s 10.244.2.4 k8s-node3 <none> <none>[root@k8s-node1 ~]# kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h48m <none>
tomcat6 NodePort 10.96.7.78 <none> 80:30055/TCP 4h15m app=tomcat6
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
扩容了多份，所有无论访问哪个node的指定端口，都可以访问到tomcat6
http://192.168.56.101:30055/
http://192.168.56.102:30055/
缩容：kubectl scale --replicas=2 deployment tomcat6
[root@k8s-node1 ~]# kubectl scale --replicas=1 deployment tomcat6
deployment.apps/tomcat6 scaled[root@k8s-node1 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
tomcat6-5f7ccf4cb9-clcpr 1/1 Running 0 4h32m 10.244.1.2 k8s-node2 <none> <none>[root@k8s-node1 ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tomcat6-5f7ccf4cb9-clcpr 1/1 Running 0 4h33mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h58m
service/tomcat6 NodePort 10.96.7.78 <none> 80:30055/TCP 4h24mNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tomcat6 1/1 1 1 4h51mNAME DESIRED CURRENT READY AGE
replicaset.apps/tomcat6-5f7ccf4cb9 1 1 1 4h51m- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
4、以上操作的yaml获取
参照k8s细节
5、删除
kubectl get all
[root@k8s-node1 ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tomcat6-5f7ccf4cb9-clcpr 1/1 Running 0 4h33mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5h58m
service/tomcat6 NodePort 10.96.7.78 <none> 80:30055/TCP 4h24mNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tomcat6 1/1 1 1 4h51mNAME DESIRED CURRENT READY AGE
replicaset.apps/tomcat6-5f7ccf4cb9 1 1 1 4h51m
#删除deployment.apps/tomcat6
[root@k8s-node1 ~]# kubectl delete deployment.apps/tomcat6
deployment.apps "tomcat6" deleted
#查看剩余的资源
[root@k8s-node1 ~]# kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h
service/tomcat6 NodePort 10.96.7.78 <none> 80:30055/TCP 4h26m
# 此时没有了部署，但是有service，没有pod只有service也是没有对应的服务的# 查看pod信息
[root@k8s-node1 ~]# kubectl get pods
No resources found in default namespace.- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
kubectl delete deploye/nginx
kubectl delete service/nginx-service
三、docker深入
四、K8s细节
1、kubectl文档
https://kubernetes.io/zh/docs/reference/kubectl/overview/
2、资源类型
https://kubernetes.io/zh/docs/reference/kubectl/overview/#资源类型
3、格式化输出
https://kubernetes.io/zh/docs/reference/kubectl/overview/
所有
kubectl命令的默认输出格式都是人类可读的纯文本格式。要以特定格式向终端窗口输出详细信息，可以将-o或--output参数添加到受支持的kubectl命令中。语法
kubectl [command] [TYPE] [NAME] -o=<output_format>
- 1
根据
kubectl操作，支持以下输出格式：
Output format Description -o custom-columns=使用逗号分隔的自定义列列表打印表。 -o custom-columns-file=使用 `` 文件中的自定义列模板打印表。 -o json输出 JSON 格式的 API 对象 `-o jsonpath= 打印 jsonpath 表达式定义的字段 -o jsonpath-file=打印 `` 文件中 jsonpath 表达式定义的字段。 -o name仅打印资源名称而不打印任何其他内容。 -o wide以纯文本格式输出，包含任何附加信息。对于 pod 包含节点名。 -o yaml输出 YAML 格式的 API 对象。 示例
在此示例中，以下命令将单个 pod 的详细信息输出为 YAML 格式的对象：
kubectl get pod web-pod-13je7 -o yaml
- 1
请记住：有关每个命令支持哪种输出格式的详细信息，请参阅 kubectl 参考文档。
–dry-run：
–dry-run=‘none’: Must be “none”, “server”, or “client”. If client strategy, only print the object that would be
sent, without sending it. If server strategy, submit server-side request without persisting the resource.
值必须为none，server或client。如果是客户端策略，则只打印该发送对象，但不发送它。如果服务器策略，提交服务器端请求而不持久化资源。
也就是说，通过–dry-run选项，并不会真正的执行这条命令。
[root@k8s-node1 ~]# kubectl create deployment tomcat6 --image=tomcat:6.0.53-jre8 --dry-run -o yamlW0504 03:39:08.389369 8107 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
apiVersion: apps/v1
kind: Deployment
metadata:creationTimestamp: nulllabels:app: tomcat6name: tomcat6
spec:replicas: 1selector:matchLabels:app: tomcat6strategy: {}template:metadata:creationTimestamp: nulllabels:app: tomcat6spec:containers:- image: tomcat:6.0.53-jre8name: tomcatresources: {}
status: {}
[root@k8s-node1 ~]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
实际上我们也可以将这个yaml输出到文件，然后使用kubectl apply -f来应用它
#输出到tomcat6.yaml
[root@k8s-node1 ~]# kubectl create deployment tomcat6 --image=tomcat:6.0.53-jre8 --dry-run -o yaml >tomcat6.yaml
W0504 03:46:18.180366 11151 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.#修改副本数为3
[root@k8s-node1 ~]# cat tomcat6.yaml
apiVersion: apps/v1
kind: Deployment
metadata:creationTimestamp: nulllabels:app: tomcat6name: tomcat6
spec:replicas: 3 #修改副本数为3selector:matchLabels:app: tomcat6strategy: {}template:metadata:creationTimestamp: nulllabels:app: tomcat6spec:containers:- image: tomcat:6.0.53-jre8name: tomcatresources: {}
status: {}#应用tomcat6.yaml
[root@k8s-node1 ~]# kubectl apply -f tomcat6.yaml
deployment.apps/tomcat6 created
[root@k8s-node1 ~]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
查看pods：
[root@k8s-node1 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
tomcat6-7b84fb5fdc-5jh6t 1/1 Running 0 8s
tomcat6-7b84fb5fdc-8lhwv 1/1 Running 0 8s
tomcat6-7b84fb5fdc-j4qmh 1/1 Running 0 8s
[root@k8s-node1 ~]#
- 1
- 2
- 3
- 4
- 5
- 6
查看某个pod的具体信息：
[root@k8s-node1 ~]# kubectl get pods tomcat6-7b84fb5fdc-5jh6t -o yaml
- 1
apiVersion: v1
kind: Pod
metadata:creationTimestamp: "2020-05-04T03:50:47Z"generateName: tomcat6-7b84fb5fdc-labels:app: tomcat6pod-template-hash: 7b84fb5fdcmanagedFields:- apiVersion: v1fieldsType: FieldsV1fieldsV1:f:metadata:f:generateName: {}f:labels:.: {}f:app: {}f:pod-template-hash: {}f:ownerReferences:.: {}k:{"uid":"292bfe3b-dd63-442e-95ce-c796ab5bdcc1"}:.: {}f:apiVersion: {}f:blockOwnerDeletion: {}f:controller: {}f:kind: {}f:name: {}f:uid: {}f:spec:f:containers:k:{"name":"tomcat"}:.: {}f:image: {}f:imagePullPolicy: {}f:name: {}f:resources: {}f:terminationMessagePath: {}f:terminationMessagePolicy: {}f:dnsPolicy: {}f:enableServiceLinks: {}f:restartPolicy: {}f:schedulerName: {}f:securityContext: {}f:terminationGracePeriodSeconds: {}manager: kube-controller-manageroperation: Updatetime: "2020-05-04T03:50:47Z"- apiVersion: v1fieldsType: FieldsV1fieldsV1:f:status:f:conditions:k:{"type":"ContainersReady"}:.: {}f:lastProbeTime: {}f:lastTransitionTime: {}f:status: {}f:type: {}k:{"type":"Initialized"}:.: {}f:lastProbeTime: {}f:lastTransitionTime: {}f:status: {}f:type: {}k:{"type":"Ready"}:.: {}f:lastProbeTime: {}f:lastTransitionTime: {}f:status: {}f:type: {}f:containerStatuses: {}f:hostIP: {}f:phase: {}f:podIP: {}f:podIPs:.: {}k:{"ip":"10.244.2.7"}:.: {}f:ip: {}f:startTime: {}manager: kubeletoperation: Updatetime: "2020-05-04T03:50:49Z"name: tomcat6-7b84fb5fdc-5jh6tnamespace: defaultownerReferences:- apiVersion: apps/v1blockOwnerDeletion: truecontroller: truekind: ReplicaSetname: tomcat6-7b84fb5fdcuid: 292bfe3b-dd63-442e-95ce-c796ab5bdcc1resourceVersion: "46229"selfLink: /api/v1/namespaces/default/pods/tomcat6-7b84fb5fdc-5jh6tuid: 2f661212-3b03-47e4-bcb8-79782d5c7578
spec:containers:- image: tomcat:6.0.53-jre8imagePullPolicy: IfNotPresentname: tomcatresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /var/run/secrets/kubernetes.io/serviceaccountname: default-token-bxqtwreadOnly: truednsPolicy: ClusterFirstenableServiceLinks: truenodeName: k8s-node2priority: 0restartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}serviceAccount: defaultserviceAccountName: defaultterminationGracePeriodSeconds: 30tolerations:- effect: NoExecutekey: node.kubernetes.io/not-readyoperator: ExiststolerationSeconds: 300- effect: NoExecutekey: node.kubernetes.io/unreachableoperator: ExiststolerationSeconds: 300volumes:- name: default-token-bxqtwsecret:defaultMode: 420secretName: default-token-bxqtw
status:conditions:- lastProbeTime: nulllastTransitionTime: "2020-05-04T03:50:47Z"status: "True"type: Initialized- lastProbeTime: nulllastTransitionTime: "2020-05-04T03:50:49Z"status: "True"type: Ready- lastProbeTime: nulllastTransitionTime: "2020-05-04T03:50:49Z"status: "True"type: ContainersReady- lastProbeTime: nulllastTransitionTime: "2020-05-04T03:50:47Z"status: "True"type: PodScheduledcontainerStatuses:- containerID: docker://18eb0798384ea44ff68712cda9be94b6fb96265206c554a15cee28c288879304image: tomcat:6.0.53-jre8imageID: docker-pullable://tomcat@sha256:8c643303012290f89c6f6852fa133b7c36ea6fbb8eb8b8c9588a432beb24dc5dlastState: {}name: tomcatready: truerestartCount: 0started: truestate:running:startedAt: "2020-05-04T03:50:49Z"hostIP: 10.0.2.4phase: RunningpodIP: 10.244.2.7podIPs:- ip: 10.244.2.7qosClass: BestEffortstartTime: "2020-05-04T03:50:47Z"
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
命令参考
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#create
service的意义
3、Service的意义
1、部署一个nginx
kubectl create deployment nginx --image=nginx2、暴露nginx访问
kubectl expose deployment nginx --port=80 --type=NodePort统一应用访问入口，
Service管理一组Pod
防止Pod失联（服务发现）、定义一组Pod的访问策略
现在我们使用NodePort的方式暴露，这样访问每个节点的端口，都可以访问各个Pod,如果节点宕机，就会出现问题。前面我们通过命令行的方式，部署和暴露了tomcat，实际上也可以通过yaml的方式来完成这些操作。
#这些操作实际上是为了获取Deployment的yaml模板
[root@k8s-node1 ~]# kubectl create deployment tomcat6 --image=tomcat:6.0.53-jre8 --dry-run -o yaml >tomcat6-deployment.yaml
W0504 04:13:28.265432 24263 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
[root@k8s-node1 ~]# ls tomcat6-deployment.yaml
tomcat6-deployment.yaml
[root@k8s-node1 ~]#
修改“tomcat6-deployment.yaml”内容如下：
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: tomcat6name: tomcat6
spec:replicas: 3selector:matchLabels:app: tomcat6template:metadata: labels:app: tomcat6spec:containers:- image: tomcat:6.0.53-jre8name: tomcat
#部署
[root@k8s-node1 ~]# kubectl apply -f tomcat6-deployment.yaml
deployment.apps/tomcat6 configured#查看资源
[root@k8s-node1 ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tomcat6-7b84fb5fdc-5jh6t 1/1 Running 0 27m
pod/tomcat6-7b84fb5fdc-8lhwv 1/1 Running 0 27m
pod/tomcat6-7b84fb5fdc-j4qmh 1/1 Running 0 27mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 14hNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tomcat6 3/3 3 3 27mNAME DESIRED CURRENT READY AGE
replicaset.apps/tomcat6-7b84fb5fdc 3 3 3 27m
[root@k8s-node1 ~]#
kubectl expose deployment tomcat6 --port=80 --target-port=8080 --type=NodePort --dry-run -o yaml
apiVersion: v1
kind: Service
metadata:creationTimestamp: nulllabels:app: tomcat6name: tomcat6
spec:ports:- port: 80protocol: TCPtargetPort: 8080selector:app: tomcat6type: NodePort
status:loadBalancer: {}
将这段输出和“tomcat6-deployment.yaml”进行拼接，表示部署完毕并进行暴露服务：
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: tomcat6name: tomcat6
spec:replicas: 3selector:matchLabels:app: tomcat6template:metadata: labels:app: tomcat6spec:containers:- image: tomcat:6.0.53-jre8name: tomcat
---
apiVersion: v1
kind: Service
metadata:creationTimestamp: nulllabels:app: tomcat6name: tomcat6
spec:ports:- port: 80protocol: TCPtargetPort: 8080selector:app: tomcat6type: NodePort
[root@k8s-node1 ~]# kubectl apply -f tomcat6-deployment.yaml
deployment.apps/tomcat6 created
service/tomcat6 created
[root@k8s-node1 ~]# kubectl get all
NAME READY STATUS RESTARTS AGE
pod/tomcat6-7b84fb5fdc-dsqmb 1/1 Running 0 4s
pod/tomcat6-7b84fb5fdc-gbmxc 1/1 Running 0 5s
pod/tomcat6-7b84fb5fdc-kjlc6 1/1 Running 0 4sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 14h
service/tomcat6 NodePort 10.96.147.210 <none> 80:30172/TCP 4sNAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/tomcat6 3/3 3 3 5sNAME DESIRED CURRENT READY AGE
replicaset.apps/tomcat6-7b84fb5fdc 3 3 3 5s
[root@k8s-node1 ~]#
[root@k8s-node1 ~]# curl -I http://192.168.56.{100,101,102}:30172/
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"7454-1491118183000"
Last-Modified: Sun, 02 Apr 2017 07:29:43 GMT
Content-Type: text/html
Content-Length: 7454
Date: Mon, 04 May 2020 04:35:35 GMTHTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"7454-1491118183000"
Last-Modified: Sun, 02 Apr 2017 07:29:43 GMT
Content-Type: text/html
Content-Length: 7454
Date: Mon, 04 May 2020 04:35:35 GMTHTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Accept-Ranges: bytes
ETag: W/"7454-1491118183000"
Last-Modified: Sun, 02 Apr 2017 07:29:43 GMT
Content-Type: text/html
Content-Length: 7454
Date: Mon, 04 May 2020 04:35:35 GMT[root@k8s-node1 ~]#
Ingress
通过Ingress发现pod进行关联。基于域名访问
通过Ingress controller实现POD负载均衡
支持TCP/UDP 4层负载均衡和HTTP 7层负载均衡
执行“k8s/ingress-controller.yaml”
[root@k8s-node1 k8s]# kubectl apply -f ingress-controller.yaml
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
daemonset.apps/nginx-ingress-controller created
service/ingress-nginx created
[root@k8s-node1 k8s]#
[root@k8s-node1 k8s]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default tomcat6-7b84fb5fdc-dsqmb 1/1 Running 0 16m
default tomcat6-7b84fb5fdc-gbmxc 1/1 Running 0 16m
default tomcat6-7b84fb5fdc-kjlc6 1/1 Running 0 16m
ingress-nginx nginx-ingress-controller-9q6cs 0/1 ContainerCreating 0 40s
ingress-nginx nginx-ingress-controller-qx572 0/1 ContainerCreating 0 40s
kube-system coredns-546565776c-9sbmk 1/1 Running 1 14h
kube-system coredns-546565776c-t68mr 1/1 Running 1 14h
kube-system etcd-k8s-node1 1/1 Running 1 14h
kube-system kube-apiserver-k8s-node1 1/1 Running 1 14h
kube-system kube-controller-manager-k8s-node1 1/1 Running 1 14h
kube-system kube-flannel-ds-amd64-5xs5j 1/1 Running 2 13h
kube-system kube-flannel-ds-amd64-6xwth 1/1 Running 2 14h
kube-system kube-flannel-ds-amd64-fvnvx 1/1 Running 1 13h
kube-system kube-proxy-7tkvl 1/1 Running 1 13h
kube-system kube-proxy-mvlnk 1/1 Running 2 13h
kube-system kube-proxy-sz2vz 1/1 Running 1 14h
kube-system kube-scheduler-k8s-node1 1/1 Running 1 14h
[root@k8s-node1 k8s]#
这里master节点负责调度，具体执行交给node2和node3来完成，能够看到它们正在下载镜像
apiVersion: extensions/v1beta1
kind: Ingress
metadata:name: web
spec:rules:- host: tomcat6.kubenetes.comhttp:paths: - backend: serviceName: tomcat6servicePort: 80
[root@k8s-node1 k8s]# touch ingress-tomcat6.yaml
#将上面的规则，添加到ingress-tomcat6.yaml文件中
[root@k8s-node1 k8s]# vi ingress-tomcat6.yaml [root@k8s-node1 k8s]# kubectl apply -f ingress-tomcat6.yaml
ingress.extensions/web created
[root@k8s-node1 k8s]#
192.168.56.102 tomcat6.kubenetes.com
测试: http://tomcat6.kubenetes.com/
安装kubernetes可视化界面——DashBoard
$ kubectl appy -f kubernetes-dashboard.yaml
默认DashBoard只能集群内部访问，修改Service为NodePort类型，暴露到外部
kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kube-system
spec:type: NodePortports:- port: 443targetPort: 8443nodePort: 3001selector:k8s-app: kubernetes-dashboard
$ kubectl create serviceaccount dashboar-admin -n kube-sysem
$ kubectl create clusterrolebinding dashboar-admin --clusterrole=cluter-admin --serviceaccount=kube-system:dashboard-admin
$ kubectl describe secrets -n kube-system $( kubectl -n kube-system get secret |awk '/dashboard-admin/{print $1}' )
kubesphere
默认的dashboard没啥用，我们用kubesphere可以打通全部的devops链路，kubesphere集成了很多套件，集群要求比较高
https://kubesphere.io
kuboard也很不错，集群要求不高
https://kuboard.cn/support/
1、简洁
2、安装前提提交
1、安装helm（master节点执行）
helm是kubernetes的包管理器。包管理器类似于在Ubuntu中使用的apt，centos中的yum或者python中的pip一样，能够快速查找，下载和安装软件包。Helm有客户端组件helm和服务端组件Tiller组成，能够将一组K8S资源打包统一管理，是查找、共享和使用为Kubernetes构建的软件的最佳方式。
1）安装
curl -L https://git.io/get_helm.sh|bash
- 1
由于被墙的原因，使用我们给定的get_helm.sh。
[root@k8s-node1 k8s]# ll
total 68
-rw-r--r-- 1 root root 7149 Feb 27 01:58 get_helm.sh
-rw-r--r-- 1 root root 6310 Feb 28 05:16 ingress-controller.yaml
-rw-r--r-- 1 root root 209 Feb 28 13:18 ingress-demo.yml
-rw-r--r-- 1 root root 236 May 4 05:09 ingress-tomcat6.yaml
-rwxr--r-- 1 root root 15016 Feb 26 15:05 kube-flannel.yml
-rw-r--r-- 1 root root 4737 Feb 26 15:38 kubernetes-dashboard.yaml
-rw-r--r-- 1 root root 3841 Feb 27 01:09 kubesphere-complete-setup.yaml
-rw-r--r-- 1 root root 392 Feb 28 11:33 master_images.sh
-rw-r--r-- 1 root root 283 Feb 28 11:34 node_images.sh
-rw-r--r-- 1 root root 1053 Feb 28 03:53 product.yaml
-rw-r--r-- 1 root root 931 May 3 10:08 Vagrantfile
[root@k8s-node1 k8s]# sh get_helm.sh
Downloading https://get.helm.sh/helm-v2.16.6-linux-amd64.tar.gz
Preparing to install helm and tiller into /usr/local/bin
helm installed into /usr/local/bin/helm
tiller installed into /usr/local/bin/tiller
Run 'helm init' to configure helm.
[root@k8s-node1 k8s]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
2）验证版本
helm version
- 1
3）创建权限（master执行）
创建helm-rbac.yaml，写入如下内容
apiVersion: v1
kind: ServiceAccount
metadata:name: tillernamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: tiller
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: cluster-admin
subjects:- kind: ServiceAccountname: kubernetes-dashboardnamespace: kube-system
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
应用配置：
[root@k8s-node1 k8s]# kubectl apply -f helm-rbac.yaml
serviceaccount/tiller created
clusterrolebinding.rbac.authorization.k8s.io/tiller created
[root@k8s-node1 k8s]#
- 1
- 2
- 3
- 4
2、安装Tilller（Master执行）
1、初始化
[root@k8s-node1 k8s]# helm init --service-account=tiller --tiller-image=sapcc/tiller:v2.16.3 --history-max 300
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /root/.helm.Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation see: https://v2.helm.sh/docs/securing_installation/
[root@k8s-node1 k8s]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
–tiller-image 指定镜像，否则会被墙，等待节点上部署的tiller完成即可。
[root@k8s-node1 k8s]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-546565776c-9sbmk 1/1 Running 3 23h
coredns-546565776c-t68mr 1/1 Running 3 23h
etcd-k8s-node1 1/1 Running 3 23h
kube-apiserver-k8s-node1 1/1 Running 3 23h
kube-controller-manager-k8s-node1 1/1 Running 3 23h
kube-flannel-ds-amd64-5xs5j 1/1 Running 4 22h
kube-flannel-ds-amd64-6xwth 1/1 Running 5 23h
kube-flannel-ds-amd64-fvnvx 1/1 Running 4 22h
kube-proxy-7tkvl 1/1 Running 3 22h
kube-proxy-mvlnk 1/1 Running 4 22h
kube-proxy-sz2vz 1/1 Running 3 23h
kube-scheduler-k8s-node1 1/1 Running 3 23h
kubernetes-dashboard-975499656-jxczv 0/1 ImagePullBackOff 0 7h45m
tiller-deploy-8cc566858-67bxb 1/1 Running 0 31s
[root@k8s-node1 k8s]#
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
查看集群的所有节点信息：
kubectl get node -o wide
- 1
[root@k8s-node1 k8s]# kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k8s-node1 Ready master 23h v1.17.3 10.0.2.15 <none> CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://19.3.8
k8s-node2 Ready <none> 22h v1.17.3 10.0.2.4 <none> CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://19.3.8
k8s-node3 Ready <none> 22h v1.17.3 10.0.2.5 <none> CentOS Linux 7 (Core) 3.10.0-957.12.2.el7.x86_64 docker://19.3.8
[root@k8s-node1 k8s]#
- 1
- 2
- 3
- 4
- 5
- 6
2、测试
helm install stable/nginx-ingress --name nginx-ingress
- 1
最小化安装 KubeSphere
若集群可用 CPU > 1 Core 且可用内存 > 2 G，可以使用以下命令最小化安装 KubeSphere：
kubectl apply -f https://raw.githubusercontent.com/kubesphere/ks-installer/master/kubesphere-minimal.yaml
- 1
提示：若您的服务器提示无法访问 GitHub，可将 kubesphere-minimal.yaml 或 kubesphere-complete-setup.yaml 文件保存到本地作为本地的静态文件，再参考上述命令进行安装。
- 查看滚动刷新的安装日志，请耐心等待安装成功。
$ kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f
- 1
说明：安装过程中若遇到问题，也可以通过以上日志命令来排查问题
【谷粒商城】集群篇-k8s(4/4)相关推荐
- 谷粒商城 集群篇 (六) --------- MySQL 集群
目录 前言 一.MySQL 集群原理 1. MySQL-MMM 2. MHA 3. InnoDB Cluster 二.Docker 安装模拟 MySQL 主从复制集群 1. 创建 Master 实例 ...
- 谷粒商城 集群篇 (四) --------- KubeSphere
目录 一.简介 二.安装 1. 前提条件 2. 安装前提环境 ① 安装 helm (master 节点执行) ② 安装 Tiller (master 执行) ③ 安装 OpenEBS (master ...
- 谷粒商城 集群篇 (五) --------- DevOps
目录 一.项目开发需要考虑的维度 二.什么是 DevOps ? 三.什么是 CI&CD 四.落地方案 五.基于 Spring Boot 项目构建流水线 1. 流水线概览 2. 创建凭证 ① 凭 ...
- 谷粒商城集群篇爬坑笔记--Gitee拉取项目报错、项目target文件不存在（部分项目不全）、SonarQube报错
Gitee拉取项目报错 问题描述 Gitee拉取可能会报错: ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitExcep ...
- 谷粒商城电商项目 高可用集群篇
更多视频,JAVA收徒 QQ:987115885谷粒商城电商项目 高可用集群篇339.k8s-简介.mp4340.k8s-架构原理&核心概念.mp4341.k8s-集群搭建-环境准备.mp43 ...
- 高可用集群篇（五）-- K8S部署微服务
高可用集群篇(五)-- K8S部署微服务 一.K8S有状态服务 1.1 什么是有状态服务 1.2 k8s部署MySQL 1.2.1 创建MySQL主从服务 1.2.2 测试主从配置 1.2.3 k8s ...
- PostgreSQL集群篇——PostgreSQL的配置文件解析
PostgreSQL集群篇--postgresql.conf配置文件解析 正文 日常中我们进行安装PostgreSQL后都需要对其进行配置基础配置,以便其能有效发挥出服务器的性能,下面是我进行整理后的 ...
- 谷粒商城-分布式高级篇【业务编写】
谷粒商城-分布式基础篇[环境准备] 谷粒商城-分布式基础[业务编写] 谷粒商城-分布式高级篇[业务编写]持续更新 谷粒商城-分布式高级篇-ElasticSearch 谷粒商城-分布式高级篇-分布式锁与 ...
- 谷粒商城--认证中心--高级篇笔记八
谷粒商城–认证中心–高级篇笔记八 1. 环境搭建 1.1 新建模块gulimall-auth-server 1.2 pom文件 上面没选好直接复制下面的pom文件,记得排除gulimall-commo ...
- [原创]分布式系统之缓存的微观应用经验谈（三）【数据分片和集群篇】
分布式系统之缓存的微观应用经验谈(三)[数据分片和集群篇] 前言 近几个月一直在忙些琐事,几乎年后都没怎么闲过.忙忙碌碌中就进入了2018年的秋天了,不得不感叹时间总是如白驹过隙,也不知道收获了什么和 ...
最新文章
- C#3.0 新特性系列(6) Extension Methods
- Linux网络编程——I/O复用之select详解
- vue项目前端服务器,【前端技术】vue自动部署项目到服务器
- php连接数据库navicat,navicat数据库如何连接php
- gis里创建要素面板怎么打开_周末技术流 | 基于GIS的地形可视化分析
- 有序列表ol与无序列表ul用法
- linux 适配电脑内核,Linux内核实践 如何添加网络协议[三]：实现 -电脑资料
- 一、计算二进制中1的个数
- CPU内部实现“加，减，乘，除”过程
- RocketMQ消息消费之长轮询
- 动软代码生成器的具体使用方法步骤
- iptv 缓存 linux,一种IPTV直播业务的卡顿检测方法及系统与流程
- 便携软件制作入门教程及实例
- 算法-狄克斯特拉算法
- (01)ORB-SLAM2源码无死角解析-(55) 闭环线程→计算Sim3:总体流程讲解ComputeSim3()
- OCJP(1Z0-851) 模拟题分析（三）
- 学编程和乐高机器人的区别
- Spring事务开启方法 传播特性 隔离界别
- stdio.h里的一些函数
- 梯形法 微积分 c语言,关于 用辛普森法和梯形法求微积分的 程序!