1. 节点信息总览

1.1 master 信息输出如下

“Capacity"和"Allocatable” 处可见，资源全部被允许被分配，即没有预留：

[root@devops-master ~]# kubectl describe nodes devops-master
Name:               devops-master
Roles:              master
#以下是给角色打的标签，架构和操作系统，等都会在里边。
Labels:             beta.kubernetes.io/arch=amd64beta.kubernetes.io/os=linuxkubernetes.io/arch=amd64kubernetes.io/hostname=devops-masterkubernetes.io/os=linuxnode-role.kubernetes.io/master=
# flannel网卡的虚拟MAC地址，也可以在ip a 中看到
Annotations:        flannel.alpha.coreos.com/backend-data: {"VtepMAC":"9e:d1:1a:e6:83:2e"}
# vxlan指 可扩展的虚拟网络flannel.alpha.coreos.com/backend-type: vxlanflannel.alpha.coreos.com/kube-subnet-manager: trueflannel.alpha.coreos.com/public-ip: 10.252.97.56kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.socknode.alpha.kubernetes.io/ttl: 0kevolumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 29 Apr 2020 16:33:15 +0800
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Conditions:Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message----             ------  -----------------                 ------------------                ------                       -------MemoryPressure   False   Wed, 12 Aug 2020 18:50:58 +0800   Wed, 29 Apr 2020 16:33:11 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory availableDiskPressure     False   Wed, 12 Aug 2020 18:50:58 +0800   Wed, 29 Apr 2020 16:33:11 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressurePIDPressure      False   Wed, 12 Aug 2020 18:50:58 +0800   Wed, 29 Apr 2020 16:33:11 +0800   KubeletHasSufficientPID      kubelet has sufficient PID availableReady            True    Wed, 12 Aug 2020 18:50:58 +0800   Wed, 29 Apr 2020 16:45:45 +0800   KubeletReady                 kubelet is posting ready status
Addresses:InternalIP:  10.252.97.56Hostname:    devops-master
#所有硬件资源
Capacity:cpu:                8ephemeral-storage:  25792732Kihugepages-1Gi:      0hugepages-2Mi:      0memory:             32765896Kipods:               110
#以下是可分配资源
Allocatable:cpu:                8ephemeral-storage:  23770581772hugepages-1Gi:      0hugepages-2Mi:      0memory:             32663496Kipods:               110
System Info:Machine ID:                 dff543df0a0c44e2962f1438f92b6868System UUID:                42277530-DD16-E8F5-B3AB-C6831B9F49FABoot ID:                    45ec85f2-0237-4d25-b684-6ec886f0c824Kernel Version:             3.10.0-514.el7.x86_64OS Image:                   CentOS Linux 7 (Core)Operating System:           linuxArchitecture:               amd64Container Runtime Version:  docker://18.6.1Kubelet Version:            v1.15.2Kube-Proxy Version:         v1.15.2
PodCIDR:                     10.244.0.0/24
Non-terminated Pods:         (8 in total)
#以下列出所有pod的信息Namespace                  Name                                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE---------                  ----                                                  ------------  ----------  ---------------  -------------  ---kube-system                coredns-bccdc95cf-vrxck                               100m (1%)     0 (0%)      70Mi (0%)        170Mi (0%)     105dkube-system                etcd-devops-master                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         105dkube-system                kube-apiserver-devops-master                          250m (3%)     0 (0%)      0 (0%)           0 (0%)         105dkube-system                kube-controller-manager-devops-master                 200m (2%)     0 (0%)      0 (0%)           0 (0%)         105dkube-system                kube-flannel-ds-amd64-bh5gv                           100m (1%)     100m (1%)   50Mi (0%)        50Mi (0%)      105dkube-system                kube-proxy-6r9sg                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         105dkube-system                kube-scheduler-devops-master                          100m (1%)     0 (0%)      0 (0%)           0 (0%)         105dmonitoring                 prometheus-operator-prometheus-node-exporter-qb48r    0 (0%)        0 (0%)      0 (0%)           0 (0%)         96d
# 以下是已分配资源
Allocated resources:(Total limits may be over 100 percent, i.e., overcommitted.)Resource           Requests    Limits--------           --------    ------cpu                750m (9%)   100m (1%)memory             120Mi (0%)  220Mi (0%)ephemeral-storage  0 (0%)      0 (0%)
Events:              <none>

1.2 node信息如下

同样查看node节点信息，可见资源同样都被分配了。

Capacity:cpu:                4ephemeral-storage:  43400496Kihugepages-1Gi:      0hugepages-2Mi:      0memory:             16247820Kipods:               110
Allocatable:cpu:                4ephemeral-storage:  43400496Kihugepages-1Gi:      0hugepages-2Mi:      0memory:             16247820Kipods:               110

说明：下文会修改这个节点的cgroup资源限制

2. 配置docker的 cgroup驱动

确认docker驱动

# docker info | grep "Cgroup Driver"
Cgroup Driver: cgroupfs

如果不是 cgroupfs，则可以通过以下方法配置

# vim /etc/docker/daemon.json
{"exec-opts": ["native.cgroupdriver=cgroupfs"],
.............
}

3. 配置kubelete的cgroup驱动

3.1 配置文件

/var/lib/kubelet/kubeadm-flags.env

作用：
用来为Kube组件和System进程预留资源，从而保证当节点出现满负荷时也能保证Kube和System进程有足够的资源。

3.2 默认配置

KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.1"

参数说明

Node Capacity：是Node的所有硬件资源

kube-reserved：是给kube组件预留的资源

system-reserved：是给System进程预留的资源

eviction-threshold：驱逐阈值

allocatable：可配置值

节点上可配置值 = 总量 - kube组件预留值 - 系统预留值 - 驱逐阈值

3.3 修改如下

KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs \--network-plugin=cni \--pod-infra-container-image=nexus.10010sh.cn/pause:3.1 \--enforce-node-allocatable=pods,kube-reserved,system-reserved \--kube-reserved-cgroup=/system.slice/kubelet.service \--system-reserved-cgroup=/system.slice \--kube-reserved=cpu=1,memory=1Gi \--system-reserved=cpu=1,memory=1Gi  \--eviction-hard=memory.available<5%,nodefs.available<10%,imagefs.available<10% \--eviction-soft=memory.available<10%,nodefs.available<15%,imagefs.available<15% \--eviction-soft-grace-period=memory.available=2m,nodefs.available=2m,imagefs.available=2m \--eviction-max-pod-grace-period=30 \--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=500Mi"

注解：

    --cgroup-driver=cgroupfs \--network-plugin=cni \--pod-infra-container-image=nexus.10010sh.cn/pause:3.1 \#开启为kube组件和系统守护进程预留资源的功能--enforce-node-allocatable=pods,kube-reserved,system-reserved \#设置k8s组件的cgroup--kube-reserved-cgroup=/system.slice/kubelet.service \#设置系统守护进程的cgroup--system-reserved-cgroup=/system.slice \# kubernetes预留--kube-reserved=cpu=1,memory=1Gi \# 系统预留--system-reserved=cpu=1,memory=1Gi  \#驱逐pod的硬限制--eviction-hard=memory.available<5%,nodefs.available<10%,imagefs.available<10% \#驱逐pod的软限制--eviction-soft=memory.available<10%,nodefs.available<15%,imagefs.available<15% \#达到驱逐阈值后多久开始驱逐--eviction-soft-grace-period=memory.available=2m,nodefs.available=2m,imagefs.available=2m \#驱逐前最大等待时间--eviction-max-pod-grace-period=30 \#至少回收多少资源才停止驱逐--eviction-minimum-reclaim=memory.available=0Mi,nodefs.available=500Mi,imagefs.available=500Mi"

3.4 修改kubelet 启动文件

[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/[Service]
ExecStart=/usr/bin/kubelet
#添加如下两行
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/cpuset/system.slice/kubelet.service
ExecStartPre=/bin/mkdir -p /sys/fs/cgroup/hugetlb/system.slice/kubelet.service
Restart=always
StartLimitInterval=0
RestartSec=10[Install]
WantedBy=multi-user.target

4. 重启服务查看结果

重启服务
如果修改了docker则需重启docker
重启kubelet
查看修改结果

Capacity:cpu:                4ephemeral-storage:  43400496Kihugepages-1Gi:      0hugepages-2Mi:      0memory:             16247820Kipods:               110
Allocatable:cpu:                2ephemeral-storage:  43400496Kihugepages-1Gi:      0hugepages-2Mi:      0memory:             13658395636pods:               110

node节点的资源限制相关推荐

K8S集群中Node节点资源不足导致Pod无法运行的故障排查思路
K8S集群中Node节点资源不足导致Pod无法运行的故障排查思路文章目录 K8S集群中Node节点资源不足导致Pod无法运行的故障排查思路 1.Node节点资源不足可能会产生的故障 2.Node节点 ...
OpenShift 4 - 使用Prometheus监控Node节点
<OpenShift 4.x HOL教程汇总> 说明:本文已经在OpenShift 4.8环境中验证文章目录对比多个节点资源使用情况查看单个节点的使用情况查看节点包括的镜像操作节 ...
oracle registers,【案例】Oracle RAC强制删除node节点过程的详细笔记
天萃荷净 Oracle研究中心案例分析:分享一篇关于Oracle RAC节点间操作的笔记,详细介绍如何强制删除Oracle RAC中的节点. 本站文章除注明转载外,均为本站原创: 转载自love wi ...
开发测试环境 k8s node节点磁盘不足运维
开发测试环境 k8s node节点磁盘不足运维开发测试环境 k8s node节点磁盘不足运维排查服务器pod 问题解决服务器节点磁盘问题开发测试环境 k8s node节点磁盘不足运维在开发测 ...
k8s node节点重启后遇到的问题及解决
有一个node节点因为主机原因进行了重启, 主机启动之后,通过执行以下命令恢复了节点状态. systemctl start docker systemctl start kubelet 在主节点查看所 ...
在Unity中创建基于Node节点的编辑器（二）窗口序列化
孙广东 2018.5.13 csdn 的产品 , 真垃圾, 不想吐槽了, 文章保存就丢! 没办法 . 怎么不满意, 还是得继续用, 哎~~~ 第二部分在Unity中序列化基于节点的 ...
Node节点禁止调度（平滑维护）方式- cordon，drain，delete
cordon.drain和delete三个命令都会使node停止被调度,后期创建的pod不会继续被调度到该节点上,但操作的暴力程度却不一样. 一.cordon 停止调度(不可调度,临时从K8S集群隔离 ...
《Kubernetes故障篇：Kubernetes Node节点DiskPressure异常处理》
文章目录一.问题背景二.问题分析 2.1.查看问题pod日志信息 2.2.查看节点磁盘容量信息 2.3.查看系统日志信息三.解决方案 3.1.方案一 3.2.方案二总结:整理不易,如果对你有帮 ...
搭建K8S集群之node节点部署
在上一篇文章中,我们完成了K8S系列之K8S集群之Master节点部署,在这篇文章中,我们将开始部署Node节点相关的组件.在node节点上,需要部署kubelet和kube-proxy两个K8S组件 ...

node节点的资源限制

文章目录

1. 节点信息总览

1.1 master 信息输出如下

1.2 node信息如下

2. 配置docker的 cgroup驱动

3. 配置kubelete的cgroup驱动

3.1 配置文件

3.2 默认配置

3.3 修改如下

3.4 修改kubelet 启动文件

4. 重启服务查看结果

node节点的资源限制相关推荐

最新文章

热门文章