kubeadm安装k8s 1.23.5
2024-05-12 23:37:25
文章目录
- 一. 环境准备
- 1.1 配置yum阿里源
- 1.2 关闭防火墙
- 1.3 关闭selinux
- 1.4 关闭swap
- 1.5 调整内核参数及模块
- 1.6 开启ipvs
- 1.7 同步服务器时间
- 1.8 安装containerd
- 1.9 修改hostname和hosts
- 二. 安装k8s
- 2.1 安装 kubelet、kubeadm、kubectl
- 2.2 初始化k8s集群
- 2.2.1 master节点（二选一）
- 2.2.2 node节点
- 2.3 安装Calico网络插件（master节点执行）
- 三. 安装其他工具
- 3.1 持久化存储 nfs
- 3.1.1 搭建NFS Server（随便安装那个节点，现在安装master节点）
- 3.1.2 安装NFS客户端(所有node节点)
- 3.1.3 安装nfs-client-provisioner
- 3.2 安装helm
- 3.2.1 下载
- 3.2.2 安装
- 3.2.3 添加几个repo
一. 环境准备
1.1 配置yum阿里源
yum -y install wget
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache
1.2 关闭防火墙
# 查看防火墙状态
firewall-cmd --state
# 临时停止防火墙
systemctl stop firewalld.service
# 禁止防火墙开机启动
systemctl disable firewalld.service
1.3 关闭selinux
# 查看selinux状态
getenforce
# 临时关闭selinux
setenforce 0
# 永久关闭selinux
sed -i 's/^ *SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
1.4 关闭swap
# 临时关闭swap
swapoff -a
# 永久关闭swap
sed -i.bak '/swap/s/^/#/' /etc/fstab
# 查看
free -g
1.5 调整内核参数及模块
加载所需内核模块
cat <<EOF> /etc/modules-load.d/k8s.conf
br_netfilter
EOFcat <<EOF> /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOFmodprobe overlay
modprobe br_netfilter
设置必需的 sysctl 参数，允许iptables检查桥接流量，这些参数在重新启动后仍然存在
cat <<EOF> /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF# 应用 sysctl 参数而无需重新启动
sudo sysctl --system
1.6 开启ipvs
不开启
ipvs将会使用iptables进行数据包转发，但是效率低，所以推荐开通ipvs，使用
cat <<EOF> /etc/sysconfig/modules/ipvs.modules
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF# 加载模块
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4# 安装了ipset软件包
yum install ipset -y# 安装管理工具ipvsadm
yum install ipvsadm -y
1.7 同步服务器时间
yum install chrony -y
systemctl enable chronyd
systemctl start chronyd[root@master ~]# chronyc sources
210 Number of sources = 4
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- ntp.wdc1.us.leaseweb.net 2 9 201 329 -8125us[-8125us] +/- 264ms
^- ntp5.flashdance.cx 2 9 373 189 -43ms[ -43ms] +/- 223ms
^+ time.cloudflare.com 3 8 377 197 +38ms[ +38ms] +/- 121ms
^* 119.28.183.184 2 8 155 30m -8460us[ -13ms] +/- 67ms[root@master ~]# date
2022年 03月 26日 星期六 15:11:32 CST
1.8 安装containerd
yum install -y yum-utils device-mapper-persistent-data lvm2yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo# 查看最新版本
yum list containerd --showduplicates | sort -ryum install containerd -y
# 安装了`containerd.io-1.5.11-3.1.el7.x86_64`containerd config default > /etc/containerd/config.toml
systemctl start containerd
systemctl enable containerd
配置
# 修改cgroups为systemd
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml# 修改基础设施镜像
sed -i 's#sandbox_image = "k8s.gcr.io/pause:3.5"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"#' /etc/containerd/config.tomlsystemctl daemon-reload
systemctl restart containerd
安装 CRI 客户端 crictl
选择版本 https://github.com/kubernetes-sigs/cri-tools/releases/
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.23.0/crictl-v1.23.0-linux-amd64.tar.gz
tar zxvf crictl-v1.23.0-linux-amd64.tar.gz -C /usr/local/bincat <<EOF> /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF# 验证是否可用
crictl pull nginx:alpine
crictl images
crictl rmi nginx:alpine
1.9 修改hostname和hosts
修改hostname
# master节点
hostnamectl set-hostname master
# node1节点
hostnamectl set-hostname node1
# node2节点
hostnamectl set-hostname node2
添加hosts
cat <<EOF> /etc/hosts
192.168.4.27 master
192.168.4.28 node1
192.168.4.29 node2
EOF
二. 安装k8s
2.1 安装 kubelet、kubeadm、kubectl
添加kubernetes源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpghttp://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
然后安装 kubeadm、kubelet、kubectl
# 查看版本，最新版 1.23.5-0
yum list kubeadm --showduplicates | sort -ryum install -y kubelet-1.23.5-0 kubectl-1.23.5-0 kubeadm-1.23.5-0[root@master ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:57:37Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
修改kubelet配置
cat <<EOF> /etc/sysconfig/kubelet
KUBELET_KUBEADM_ARGS="--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF
启动kubelet服务，并设置开机自启
systemctl start kubelet
systemctl enable kubelet
2.2 初始化k8s集群
2.2.1 master节点（二选一）
1. 通过配置文件初始化：
kubeadm config print init-defaults > kubeadm.yaml修改为
cat <<EOF> kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: 192.168.4.27 # apiserver 节点内网IPbindPort: 6443
nodeRegistration:criSocket: /run/containerd/containerd.sock # 修改为containerdimagePullPolicy: IfNotPresentname: mastertaints:- effect: NoSchedulekey: node-role.kubernetes.io/master
---
apiServer:timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:type: CoreDNS # dns类型 type: CoreDNS
etcd:local:dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers # 修改这个镜像能下载
kind: ClusterConfiguration
kubernetesVersion: 1.23.5 # k8s版本
networking:dnsDomain: cluster.localpodSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs # kube-proxy 模式
EOFkubeadm init --config kubeadm.yaml
2. 直接初始化：
kube-proxy模式是iptables，可以通过kubectl edit configmap kube-proxy -n kube-system修改
kubeadm init \
--kubernetes-version v1.23.5 \
--apiserver-advertise-address 192.168.4.27 \
--control-plane-endpoint master \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr 10.244.0.0/16 \
--cri-socket /run/containerd/containerd.sock
--kubernetes-version：指定的版本--apiserver-advertise-address：K8S主节点的地址--pod-network-cidr：pod的网络IP范围
如果您的网络运行在
192.168.*.*，需要将pod-network-cidr设置为10.0.0.0/16；
如果您的网络是10.0.*.*使用192.168.0.0/16，此时使用calico网络（如果设置错了部署calico 网络插件后coredns也运行不起来，会报错coredns Failed to list *v1.Endpoints，该错误解决办法参考https://blog.csdn.net/u011663005/article/details/87937800）：
--image-repository：指定下载源
复制config文件
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
2.2.2 node节点
kubeadm join master:6443 --token f6e3hv.uk6ctfgehstt92jw \--discovery-token-ca-cert-hash sha256:9962caed607e31de7b93732347c1ac681f216c290e6b35f91f3f5d67cd12cbcf
2.3 安装Calico网络插件（master节点执行）
mkdir -p /root/i && cd /root/i# 下载
curl https://docs.projectcalico.org/manifests/calico.yaml -o /root/i/calico.yaml查看一下版本`v3.22.2`，如果不是替换不生效# 修改镜像
sed -i 's#docker.io/calico/cni:v3.22.2#registry.cn-shanghai.aliyuncs.com/wanfei/cni:v3.22.2#' /root/i/calico.yaml
sed -i 's#docker.io/calico/pod2daemon-flexvol:v3.22.2#registry.cn-shanghai.aliyuncs.com/wanfei/pod2daemon-flexvol:v3.22.2#' /root/i/calico.yaml
sed -i 's#docker.io/calico/node:v3.22.2#registry.cn-shanghai.aliyuncs.com/wanfei/node:v3.22.2#' /root/i/calico.yaml
sed -i 's#docker.io/calico/kube-controllers:v3.22.2#registry.cn-shanghai.aliyuncs.com/wanfei/kube-controllers:v3.22.2#' /root/i/calico.yaml# 执行
kubectl apply -f /root/i/calico.yaml
等几分钟
[root@master i]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-57845f44bb-tpvbr 1/1 Running 0 79s
calico-node-fpfxj 1/1 Running 0 79s
calico-node-qcvqx 1/1 Running 0 79s
calico-node-r4gsf 1/1 Running 0 79s
coredns-6d8c4cb4d-7bclr 1/1 Running 0 29m
coredns-6d8c4cb4d-djwxf 1/1 Running 0 29m
etcd-master 1/1 Running 0 29m
kube-apiserver-master 1/1 Running 0 29m
kube-controller-manager-master 1/1 Running 0 29m
kube-proxy-pjkmd 1/1 Running 0 7m35s
kube-proxy-snb84 1/1 Running 0 7m46s
kube-proxy-tp7wm 1/1 Running 0 29m
kube-scheduler-master 1/1 Running 0 29m[root@master i]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 29m v1.23.5
node1 Ready <none> 8m4s v1.23.5
node2 Ready <none> 7m53s v1.23.5
三. 安装其他工具
3.1 持久化存储 nfs
3.1.1 搭建NFS Server（随便安装那个节点，现在安装master节点）
yum -y install nfs-utils rpcbind#分配权限
mkdir /nfsdata && chmod 666 /nfsdata && chown nfsnobody /nfsdata# 配置挂载
cat <<EOF> /etc/exports
/nfsdata *(rw,no_root_squash,no_all_squash,sync)
EOF# 启动
systemctl start rpcbind.service
systemctl enable rpcbind.service
systemctl start nfs.service
systemctl enable nfs.service
3.1.2 安装NFS客户端(所有node节点)
如果不安装，使用
StorageClass的nfs-client的自动配置程序，我们也叫它Provisioner所在的node节点就会一直ContainerCreating
[root@master nfs-client]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nfs-client-provisioner-798cfd7476-zrndd 0/1 ContainerCreating 0 3m53s <none> node1 <none> <none>
安装
yum -y install nfs-utils rpcbindsystemctl start rpcbind.service
systemctl enable rpcbind.service
systemctl start nfs.service
systemctl enable nfs.service[root@node1 ~]# showmount -e 192.168.4.27
Export list for 192.168.4.27:
/nfsdata *
3.1.3 安装nfs-client-provisioner
设置
StorageClass，自动生成PV
nfs-rbac.yaml
cat <<EOF> nfs-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:name: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default #根据实际环境设定namespace,下面类同
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: nfs-client-provisioner-runner
rules:- apiGroups: [""]resources: ["persistentvolumes"]verbs: ["get", "list", "watch", "create", "delete"]- apiGroups: [""]resources: ["persistentvolumeclaims"]verbs: ["get", "list", "watch", "update"]- apiGroups: ["storage.k8s.io"]resources: ["storageclasses"]verbs: ["get", "list", "watch"]- apiGroups: [""]resources: ["events"]verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: run-nfs-client-provisioner
subjects:- kind: ServiceAccountname: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default
roleRef:kind: ClusterRolename: nfs-client-provisioner-runnerapiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default
rules:- apiGroups: [""]resources: ["endpoints"]verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisioner
subjects:- kind: ServiceAccountname: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default
roleRef:kind: Rolename: leader-locking-nfs-client-provisionerapiGroup: rbac.authorization.k8s.io
EOF
nfs-storage.yaml
cat <<EOF> nfs-storage.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: nfs-storage
provisioner: nfs-storage #这里的名称要和provisioner配置文件中的环境变量PROVISIONER_NAME保持一致
parameters:archiveOnDelete: "true"
reclaimPolicy: Retain
EOF
nfs-provisioner.yaml
cat <<EOF> nfs-provisioner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: nfs-client-provisionerlabels:app: nfs-client-provisioner# replace with namespace where provisioner is deployednamespace: default #与RBAC文件中的namespace保持一致
spec:replicas: 1selector:matchLabels:app: nfs-client-provisionerstrategy:type: Recreateselector:matchLabels:app: nfs-client-provisionertemplate:metadata:labels:app: nfs-client-provisionerspec:serviceAccountName: nfs-client-provisionercontainers:- name: nfs-client-provisioner#image: quay.io/external_storage/nfs-client-provisioner:latest#这里特别注意，在k8s-1.20以后版本中使用上面提供的包，并不好用，这里我折腾了好久，才解决，后来在官方的github上，别人提的问题中建议使用下面这个包才解决的，我这里是下载后，传到我自已的仓库里#easzlab/nfs-subdir-external-provisioner:v4.0.2image: registry.cn-shanghai.aliyuncs.com/wanfei/nfs-subdir-external-provisioner:v4.0.2volumeMounts:- name: nfs-client-rootmountPath: /persistentvolumesenv:- name: PROVISIONER_NAMEvalue: nfs-storage #provisioner名称,请确保该名称与 nfs-StorageClass.yaml文件中的provisioner名称保持一致- name: NFS_SERVERvalue: 192.168.4.27 #NFS Server IP地址- name: NFS_PATHvalue: "/nfsdata" #NFS挂载卷volumes:- name: nfs-client-rootnfs:server: 192.168.4.27 #NFS Server IP地址path: "/nfsdata" #NFS 挂载卷
EOF
安装
kubectl apply -f .[root@master nfs-client]# kubectl get pods | grep nfs-client
nfs-client-provisioner-777fbf8b55-2ptbm 1/1 Running 0 34s
设置默认的StorageClass(有default)
kubectl patch storageclass nfs-storage -p '{ "metadata" : { "annotations" :{"storageclass.kubernetes.io/is-default-class": "true"}}}'[root@master ~]# kubectl get sc | grep nfs-storage
nfs-storage (default) nfs-storage Retain Immediate false 71s# 取消default,值为"false"
kubectl patch storageclass nfs-storage -p '{ "metadata" : { "annotations" :{"storageclass.kubernetes.io/is-default-class": "false"}}}'
参考 https://blog.csdn.net/m0_48898914/article/details/121752973
3.2 安装helm
3.2.1 下载
下载地址 https://github.com/helm/helm/releases
[root@master helm]# wget https://get.helm.sh/helm-v3.8.1-linux-amd64.tar.gz
服务器下载速度太慢，可以科学上网下载好了上传到服务器，安装
lrzszhttps://blog.csdn.net/qq_22356995/article/details/104071562
解压
[root@master helm]# tar -xvf helm-v3.8.1-linux-amd64.tar.gz
linux-amd64/
linux-amd64/helm
linux-amd64/README.md
linux-amd64/LICENSE
3.2.2 安装
将helm移到/usr/local/bin目录
[root@master helm]# mv linux-amd64/helm /usr/local/bin
[root@master helm]# helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}
3.2.3 添加几个repo
helm repo add apphub https://apphub.aliyuncs.com
helm repo add stable https://charts.helm.sh/stable
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update# 例如搜索redis chart
[root@master helm]# helm search repo redis
NAME CHART VERSION APP VERSION DESCRIPTION
apphub/prometheus-redis-exporter 3.2.2 1.3.4 Prometheus exporter for Redis metrics
apphub/redis 10.5.3 5.0.7 Open source, advanced key-value store. It is of...
apphub/redis-cache 0.5.0 4.0.12-alpine A pure in-memory redis cache, using statefulset...
apphub/redis-ha 4.3.3 5.0.6 Highly available Kubernetes implementation of R...
apphub/redis-operator 1.0.0 Redis Operator provides high availability redis...
apphub/redispapa 0.0.1 0.0.1 利用redis的info信息对redis的使用情况进行监控的一...
bitnami/redis 16.6.0 6.2.6 Redis(TM) is an open source, advanced key-value...
bitnami/redis-cluster 7.4.1 6.2.6 Redis(TM) is an open source, scalable, distribu...
stable/prometheus-redis-exporter 3.5.1 1.3.4 DEPRECATED Prometheus exporter for Redis metrics
stable/redis 10.5.7 5.0.7 DEPRECATED Open source, advanced key-value stor...
stable/redis-ha 4.4.6 5.0.6 DEPRECATED - Highly available Kubernetes implem...
stable/sensu 0.2.5 0.28 DEPRECATED Sensu monitoring framework backed by...
apphub/codis 3.2 3.2 A Helm chart for Codis
kubeadm安装k8s 1.23.5相关推荐
- unutun21.04安装k8s v1.23.1（一）
unutun21.04安装k8s v1.23.1 1. 环境初始化 2. 安装docker 2.1 安装依赖 2.2 安装gpg证书 2.3 写入软件源信息 2.4 更新并安装Docker-ce 2. ...
- 用Kubeadm安装K8s后，kube-flannel-ds一直CrashLoopBackOff
2019独角兽企业重金招聘Python工程师标准>>> 如果使用Kubeadm安装K8s集群,在安装flannel网络插件后,发现pod: kube-flannel-ds 一直是Cr ...
- (亲测无坑)Centos7.x使用kubeadm安装K8s集群1.15.0版本
基础环境配置 三台Centos7.x的服务器,主节点 cpu >=2,node节点>=1 注:(上述cpu为最低配置,否则集群安装部署会报错,无法启动,对其他硬件无硬性要求) 以下操作若无 ...
- CentOS7安装K8S V1.23.3
一.系统准备 查看系统版本 [root@localhost docker]# cat /etc/centos-release CentOS Linux release 7.9.2009 (Core) ...
- 局域网使用kubeadm安装k8s
主机列表: ip 主机名 节点 cpu 内存 192.168.23.100 k8smaster master 2核 2G 192.168.23.101 k8snode01 node 2核 2G 192 ...
- 【k8s系列】vmware中ubuntu22.04使用kubeadm安装k8s集群(1.24.4 or higher)(cka考试环境搭建)
文章目录 部署环境准备 环境修改 修改hostname(各修改各的) 修改hosts文件(master 和所有node ) 禁用swap(master 和所有node) 修改内核参数(master 和 ...
- centos7下使用kubeadm安装k8s
kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具.这个工具能通过两条指令完成一个kubernetes集群的部署. 在开始之前,部署Kubernetes集群机器需要满足以下几 ...
- kubeadm安装K8S单master双节点集群
宿主机: master:172.16.40.97 node1:172.16.40.98 node2:172.16.40.99 # 一.k8s初始化环境:(三台宿主机) 关闭防火墙和selinux sy ...
- 使用kubeadm安装k8s集群故障处理三则
最近在作安装k8s集群,测试了几种方法,最终觉得用kubeadm应该最规范. 限于公司特别的网络情况,其安装比网上不能访问google的情况还要艰难. 慢慢积累经验吧. 今天遇到的三则故障记下来作参考 ...
最新文章
- 操作系统读写者问题实验报告_操作系统知识点总结
- R语言安装.tar.gz包
- PyTorch LSTM，batch_first=True对初始化h0和c0的影响
- pjsip视频通信开发(上层应用)之拨号界面整体界面功能实现
- php索引是什么意思,为什么我的php一直说 没索引！下面是代码
- 编译器编译报错时aka是什么意思？（also known as）
- OAuth2.0学习（1-1）OAuth2.0是什么？
- 实现一个可以用web配置的防火墙
- python工资一般多少大专学历-这个面试官真霸气 不收大专程序员
- hive当前日期超前_微博数仓数据延时优化方案
- 【转载】C++ function、bind和lambda表达式
- 关于Android 微信APP支付开发中遇到的问题
- VB中常用的的ASCII码chr()对应表
- 全面解析云智慧数据中心统一运管解决方案
- hive建表语句 增加字段、分区基础操作
- python实现商品进销存管理系统
- Excel表格实现下拉复选框多选
- RK3588 Android12开机log分析
- 将负数转换为整数（绝对值函数）
- 如何用photoshop做24色环_Photoshop绘制超逼真的色轮/色环配色表效果图