k8s(一)、 1.9.0高可用集群本地离线部署记录
一、部署说明
1.节点
master1: IP:192.168.0.170/24 hostname:171
master2: IP:192.168.0.171/24 hostname:172
VIP:192.168.0.169/24
2.工具版本:
etcd-v3.1.10-linux-amd64.tar.gz
kubeadm-1.9.0-0.x86_64.rpm
kubectl-1.9.0-0.x86_64.rpm
kubelet-1.9.9-9.x86_64.rpm
kubernetes-cni-0.6.0-0.x86_64.rpm
socat-1.7.3.2-2.el7.x86_64.rpm
docker离线镜像:
etcd-amd64_v3.1.10.tar
k8s-dns-dnsmasq-nanny-amd64_v1.14.7.tar
k8s-dns-kube-dns-amd64_1.14.7.tar
k8s-dns-sidecar-amd64_1.14.7.tar
kube-apiserver-amd64_v1.9.0.tar
kube-controller-manager-amd64_v1.9.0.tar
kube-proxy-amd64_v1.9.0.tar
kubernetes-dashboard_v1.8.1.tar
kube-scheduler-amd64_v1.9.0.tar
pause-amd64_3.0.tar
以上安装包的云盘链接随后附上.有梯子翻墙的直接装,无需离线包。
=================================================
k8s 高可用2个核心 apiserver master和etcd
apiserver master:(需高可用)集群核心,集群API接口、集群各个组件通信的中枢;集群安全控制;
etcd :(需高可用)集群的数据中心,用于存放集群的配置以及状态信息,非常重要,如果数据丢失那么集群将无法恢复;因此高可用集群部署首先就是etcd是高可用集群;
kube-scheduler:调度器 (内部自选举)集群Pod的调度中心;默认kubeadm安装情况下–leader-elect参数已经设置为true,保证master集群中只有一个kube-scheduler处于活跃状态;
kube-controller-manager: 控制器(内部自选举)集群状态管理器,当集群状态与期望不同时,kcm会努力让集群恢复期望状态,比如:当一个pod死掉,kcm会努力新建一个pod来恢复对应replicasset期望的状态;默认kubeadm安装情况下–leader-elect参数已经设置为true,保证master集群中只有一个kube-controller-manager处于活跃状态;
kubelet: 将agent node注册上apiserver
kube-proxy: 每个node上一个,负责service vip到endpoint pod的流量转发,老版本主要通过设置iptables规则实现,新版1.9基于kube-proxy-lvs 实现
二、提前准备
1.关闭防火墙,selinux,swap
systemctl stop firewalld
systemctl disable firewalld
systemctl disable firewalldsetenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config swapoff -a
echo 'swapoff -a ' >> /etc/rc.d/rc.local
2.内核开启网络支持
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.conf
若执行sysctl -p 时报错,尝试:
modprobe br_netfilter
ls /proc/sys/net/bridge
三、配置keepalived
yum -y install keepalived # 192.168.0.170配置
cat >/etc/keepalived/keepalived.conf <<EOF
global_defs {router_id LVS_k8s
}vrrp_script CheckK8sMaster {#修改为本机IPscript "curl -k https://192.168.0.170:6443"interval 3timeout 5weight 11fall 3rise 2
}vrrp_instance VI_0 {# 统一默认设置为BACKUP状态,成员会根据优先级值来自协商状态state BACKUP interface eth0virtual_router_id 80# 主节点权重最高 依次减少priority 110advert_int 1#修改为本地IPmcast_src_ip 192.168.0.170track_interface {eth0}authentication {auth_type PASSauth_pass yks888}unicast_peer {#注释掉本地IP#192.168.0.170192.168.0.171}virtual_ipaddress {192.168.0.169/24}track_script {CheckK8sMaster}
}
EOF# 192.168.0.171配置
cat >/etc/keepalived/keepalived.conf <<EOLglobal_defs {router_id LVS_k8s
}vrrp_script CheckK8sMaster {#修改为本机IPscript "curl -k https://192.168.0.171:6443"interval 3timeout 5weight 11fall 3rise 2
}vrrp_instance VI_0 {# 统一默认设置为BACKUP状态,成员会根据优先级值来自协商状态state BACKUP interface eth0virtual_router_id 80# 主节点权重最高 依次减少priority 100advert_int 1#修改为本地IPmcast_src_ip 192.168.0.171track_interface {eth0}authentication {auth_type PASSauth_pass yks888}unicast_peer {#注释掉本地IP192.168.0.170#192.168.0.171}virtual_ipaddress {192.168.0.169/24}track_script {CheckK8sMaster}
}
EOF
systemctl enable keepalived && systemctl restart keepalived
检查vip是否正常漂在master1上:
[root@170 pkgs]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000link/ether 00:0c:29:1c:5d:71 brd ff:ff:ff:ff:ff:ffinet 192.168.0.170/24 brd 192.168.0.255 scope global ens33valid_lft forever preferred_lft foreverinet 192.168.0.169/24 scope global secondary ens33valid_lft forever preferred_lft forever
四、etcd配置
1.配置环境变量:
export NODE_NAME=170 #当前部署的机器名称(随便定义,只要能区分不同机器即可)
export NODE_IP=192.168.0.170 # 当前部署的机器 IP
export NODE_IPS="192.168.0.170 192.168.0.171 " # etcd 集群所有机器 IP
# etcd 集群间通信的IP和端口
export ETCD_NODES="170"=https://192.168.0.170:2380,"171"=https://192.168.0.171:2380
2.创建ca和证书和秘钥
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfsslwget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljsonwget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
生成ETCD的TLS 秘钥和证书
为了保证通信安全,客户端(如 etcdctl) 与 etcd 集群、etcd 集群之间的通信需要使用 TLS 加密,本节创建 etcd TLS 加密所需的证书和私钥。
创建 CA 配置文件:
cat > ca-config.json <<EOF
{
"signing": {
"default": {"expiry": "87600h"
},
"profiles": {"kubernetes": {"usages": ["signing","key encipherment","server auth","client auth"],"expiry": "87600h"}
}
}
}
EOF
ca-config.json:可以定义多个 profiles,分别指定不同的过期时间、使用场景等参数;后续在签名证书时使用某个 profile;
signing:表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE;
server auth:表示 client 可以用该 CA 对 server 提供的证书进行验证;
client auth:表示 server 可以用该 CA 对 client 提供的证书进行验证;
cat > ca-csr.json <<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "System"
}
]
}
EOF
“CN”:Common Name,kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name);浏览器使用该字段验证网站是否合法;
“O”:Organization,kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group);
生成 CA 证书和私钥:
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls ca*
创建 etcd 证书签名请求:
cat > etcd-csr.json <<EOF
{"CN": "etcd","hosts": ["127.0.0.1","192.168.0.170","192.168.0.171"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "System"}]
}
EOF
hosts 字段指定授权使用该证书的 etcd 节点 IP;
每个节点IP 都要在里面 或者 每个机器申请一个对应IP的证书
生成 etcd 证书和私钥:
cfssl gencert -ca=ca.pem \-ca-key=ca-key.pem \-config=ca-config.json \-profile=kubernetes etcd-csr.json | cfssljson -bare etcdls etcd*mkdir -p /etc/etcd/ssl
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
将生成的私钥文件分发到master2上:
scp -r /etc/etcd/ssl root@192.168.0.171:/etc/etcd/
3.安装etcd
wget http://github.com/coreos/etcd/releases/download/v3.1.10/etcd-v3.1.10-linux-amd64.tar.gz
tar -xvf etcd-v3.1.10-linux-amd64.tar.gz
mv etcd-v3.1.10-linux-amd64/etcd* /usr/local/bin
mkdir -p /var/lib/etcd # 必须先创建工作目录cat > etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \\--name=${NODE_NAME} \\--cert-file=/etc/etcd/ssl/etcd.pem \\--key-file=/etc/etcd/ssl/etcd-key.pem \\--peer-cert-file=/etc/etcd/ssl/etcd.pem \\--peer-key-file=/etc/etcd/ssl/etcd-key.pem \\--trusted-ca-file=/etc/etcd/ssl/ca.pem \\--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\--initial-advertise-peer-urls=https://${NODE_IP}:2380 \\--listen-peer-urls=https://${NODE_IP}:2380 \\--listen-client-urls=https://${NODE_IP}:2379,http://127.0.0.1:2379 \\--advertise-client-urls=https://${NODE_IP}:2379 \\--initial-cluster-token=etcd-cluster-0 \\--initial-cluster=${ETCD_NODES} \\--initial-cluster-state=new \\--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF#注意,上方的环境变量一定要先配置好再进行这一步操作,在master2上export环境变量时注意修改ip
mv etcd.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd
master2同样方法安装etcd
验证服务
etcdctl \--endpoints=https://${NODE_IP}:2379 \--ca-file=/etc/etcd/ssl/ca.pem \--cert-file=/etc/etcd/ssl/etcd.pem \--key-file=/etc/etcd/ssl/etcd-key.pem \cluster-health
预期结果:
2018-04-06 20:07:41.355496 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2018-04-06 20:07:41.357436 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 57d56677d6df8a53 is healthy: got healthy result from https://192.168.0.170:2379
member 7ba40ace706232da is healthy: got healthy result from https://192.168.0.171:2379
若有失败,重新配置
systemctl stop etcdrm -Rf /var/lib/etcdrm -Rf /var/lib/etcd-clustermkdir -p /var/lib/etcdsystemctl start etcd
五.docker安装
若原来已经安装docker版本,建议卸载后安装docker-ce版本:
yum -y remove docker docker-common
基于aliyun安装docker-ce方法:
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
# Step 2: 添加软件源信息
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo# Step 3: 更新并安装 Docker-CE
sudo yum makecache fast#查看repo中所有版本的docker-ce
yum list docker-ce --showduplicates | sort -r
docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stablesudo yum -y install docker-ce-17.03.2.ce
#若提示依赖docker-ce-selinux,则先执行yum -y install docker-ce-selinux-17.03.2.ce
# Step 4: 开启Docker服务
sudo service docker start && systemctl enbale docker
六.安装kubelet、kubectl、kubeadm、kubecni
准备好提前下载好的rpm包:
[root@170 pkgs]# cd /root/k8s_images/pkgs
[root@170 pkgs]# ls
etcd-v3.1.10-linux-amd64.tar.gz kubeadm-1.9.0-0.x86_64.rpm kubectl-1.9.0-0.x86_64.rpm kubelet-1.9.9-9.x86_64.rpm kubernetes-cni-0.6.0-0.x86_64.rpm socat-1.7.3.2-2.el7.x86_64.rpm[root@170 pkgs]# yum -y install *.rpm
修改kubelet启动参数里的cgroup驱动以便兼容docker:
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
#Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"systemctl enable docker && systemctl restart docker
systemctl daemon-reload && systemctl restart kubelet
在master2上同样操作
七.kubeadm初始化
将准备好的本地的docker images镜像导入本地仓库内
[root@170 docker_images]# ls
etcd-amd64_v3.1.10.tar k8s-dns-dnsmasq-nanny-amd64_v1.14.7.tar k8s-dns-sidecar-amd64_1.14.7.tar kube-controller-manager-amd64_v1.9.0.tar kubernetes-dashboard_v1.8.1.tar pause-amd64_3.0.tar
flannel:v0.9.1-amd64.tar k8s-dns-kube-dns-amd64_1.14.7.tar kube-apiserver-amd64_v1.9.0.tar kube-proxy-amd64_v1.9.0.tar kube-scheduler-amd64_v1.9.0.tar[root@170 docker_images]# for i in `ls`;do docker load -i $i;done
config.yaml初始化配置文件:
cat <<EOF > config.yaml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:endpoints:- https://192.168.0.170:2379- https://192.168.0.171:2379caFile: /etc/etcd/ssl/ca.pem certFile: /etc/etcd/ssl/etcd.pem keyFile: /etc/etcd/ssl/etcd-key.pemdataDir: /var/lib/etcd
networking:podSubnet: 10.244.0.0/16
kubernetesVersion: 1.9.0
api:advertiseAddress: "192.168.0.170"
token: "b99a00.a144ef80536d4344"
tokenTTL: "0s"
apiServerCertSANs:
- "170"
- "171"
- 192.168.0.169
- 192.168.0.170
- 192.168.0.171
featureGates:CoreDNS: true
EOF
初始化集群
kubeadm init --config config.yaml
结果:
To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config
as root:kubeadm join --token b99a00.a144ef80536d4344 192.168.0.169:6443 --discovery-token-ca-cert-hash sha256:ebc2f64e9bcb14639f26db90288b988c90efc43828829c557b6b66bbe6d68dfa[root@170 docker_images]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
170 Ready master 5h v1.9.0 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://18.3.0
171 Ready master 4h v1.9.0 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://18.3.0
注意:提示信息中的kubeadm join那一行请保存好,后期使用kubeadm扩增集群node时,需要在新的node上使用这行命令加入集群
mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config
配置master2:
#拷贝pki 证书
mkdir -p /etc/kubernetes/pki
scp -r /etc/kubernetes/pki/ root@192.168.0.171:/etc/kubernetes/ #拷贝初始化配置
scp config.yaml root@192.168.0.171:/etc/kubernetes/#在master2上初始化
kubeadm init --config /etc/kubernetes/config.yaml
八、安装网络插件kube-router
# kube-router的daemonSet部署yaml文件地址:
# https://github.com/cloudnativelabs/kube-router/blob/master/daemonset/kubeadm-kuberouter.yaml
kubectl apply -f kubeadm-kuberouter.yaml[root@170 docker_images]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-546545bc84-j5mds 1/1 Running 2 5h
kube-system kube-apiserver-170 1/1 Running 2 5h
kube-system kube-apiserver-171 1/1 Running 2 4h
kube-system kube-controller-manager-170 1/1 Running 2 5h
kube-system kube-controller-manager-171 1/1 Running 2 4h
kube-system kube-proxy-5dg4j 1/1 Running 2 4h
kube-system kube-proxy-s6jtn 1/1 Running 2 5h
kube-system kube-router-wbxvq 1/1 Running 2 4h
kube-system kube-router-xrkbw 1/1 Running 2 4h
kube-system kube-scheduler-170 1/1 Running 2 5h
kube-system kube-scheduler-171 1/1 Running 2 4h
部署完成
默认情况下,为了保证master的稳定,master存在调度污点不会调度pod。取消master调度污点:
kubectl taint nodes --all node-role.kubernetes.io/master-
配置私有docker仓库,加速使用:
vim /lib/systemd/system/docker.service
注释#ExecStart=/usr/bin/dockerdExecStart=/usr/bin/dockerd --insecure-registry registry.yourself.com:5000
#填写自己的docker仓库地址
6.8补充:
k8s集群通过kubeadm加入新的node:
在集群初始化完成时,会提示如下信息:
To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config
as root:kubeadm join --token b99a00.a144ef80536d4344 192.168.0.169:6443 --discovery-token-ca-cert-hash sha256:ebc2f64e9bcb14639f26db90288b988c90efc43828829c557b6b66bbe6d68dfa
其中kubeadm join是为后期新的node加入集群时使用的,在公司生产上的集群部署好时此命令没有保存,且token时有实效性的。后期有node加入集群,join命令的两个参数可以在master节点上通过如下命令获取:
token:
root@yksp009027:~# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
b97h00.a144ef08526d4345 <forever> <never> authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
hash:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
7eb62962040916673ab58b0c40fb451caef7538452f5e6c8ff7719a81fdbf633
然后在新node上执行:
kubeadm join --token=$TOKEN $MASTER_IP:6443 --discovery-token-ca-cert-hash sha256:$HASH
查看node,kubectl get nodes新node已进入Ready状态。(在join前请保证node上安装好kubelet、docker、kubeadm、kubernetes-cni、socat,并启动kubelet、docker,参考上方步骤)
k8s(一)、 1.9.0高可用集群本地离线部署记录相关推荐
- Kubeadm HA 1.9 高可用 集群 本地离线部署
Kubeadm HA 1.9 高可用 集群 本地离线部署 k8s介绍 k8s 发展速度很快,目前很多大的公司容器集群都基于该项目,如京东,腾讯,滴滴,瓜子二手车,北森等等. kubernetes1.9 ...
- [K8s 1.9实践]Kubeadm 1.9 HA 高可用 集群 本地离线镜像部署
Kubeadm HA 1.9 高可用 集群 本地离线部署 k8s介绍 k8s 发展速度很快,目前很多大的公司容器集群都基于该项目,如京东,腾讯,滴滴,瓜子二手车,易宝支付,北森等等. kubernet ...
- Kubeadm 1.9 HA 高可用集群本地离线镜像部署【已验证】
k8s介绍 k8s 发展速度很快,目前很多大的公司容器集群都基于该项目,如京东,腾讯,滴滴,瓜子二手车,易宝支付,北森等等. kubernetes1.9版本发布2017年12月15日,每三个月一个迭代 ...
- Hadoop2.0高可用集群搭建【保姆级教程】
搭载Hadoop2.0高可用集群 说明 准备 下载好所需要的文件 目录准备 虚拟机网络配置(可能会在其他文章中讲到) 文件的安装 配置环境变量 环境变量的验证 关闭防火墙 配置Hadoop高可用集群 ...
- 生产环境kubeadm部署k8s(1.23)高可用集群
kubeadm部署k8s高可用集群 1.设备清单 2.各节点下载docker源 3.各节点安装docker服务并加入开机启动 4.各节点配置docker加速器并修改成k8s驱动 5.各节点重启dock ...
- k8s高可用集群_搭建高可用集群(初始化和部署keepalived)---K8S_Google工作笔记0055
技术交流QQ群[JAVA,C++,Python,.NET,BigData,AI]:170933152 然后我们来部署keepalived,搭建高可用集群. 然后我们这里需要master,155, ma ...
- 最新版Kubernetes(k8s)-v1.22.3版本高可用集群
目录 一.环境准备 二.安装Docker 三.配置环境变量 四.所有master节点安装keepalived和haproxy服务 五.部署集群 六.部署k8s的dashboard 本文采用的是etcd ...
- k8s:概念以及搭建高可用集群
一.k8s概念和架构 1.k8s概述 k8s是谷歌在2014年开源的容器化集群管理系统 使用k8s进行容器化应用部署 使用k8s利于应用扩展 k8s目标实施让容器化应用程序更加简洁高效 2.特性 (1 ...
- CapitalOne - Artifactory高可用集群的自动化部署实践
背景 本文为大家介绍Capital One如何利用自动化流水线实现Artifactory HA集群进行自动化运维.Capital One银行是美国最大的数字化银行之一,在Capital One的dev ...
最新文章
- C++读某个内存地址对应的值
- java执行器是什么_java使用Executor(执行器)管理线程
- pycharm 类型注释_后端开发使用pycharm的技巧
- explain 之 select_type
- Toast的另类应用及另类“拦截”Home键
- ubuntu 下安装 cudnn
- ubuntu下超级用户和普通用户
- hdu 1233 最小生成树
- .net System.Web.Mail发送邮件
- 利用网络媒体搭建自己就业之“桥”——卖糖葫芦北大毕业生谋职报道带来的启示...
- oracle flashback 功能,oracle 10g中开启flashback功能
- 达梦数据库可视化管理工具的配置
- Spring Cloud Gray 微服务灰度中间件
- excel换行快捷键_超实用的16个Excel快捷键,一定要收藏!
- Windows系统下GIT生成密钥和添加密钥git
- Android开发 ANR异常的解决(应用程序无响应)
- 这三个步骤让你知道WPS如何转换成WORD文档
- 笔记本电脑外接显示器完全攻略(图文说明)
- 局域网下访问自己的项目和网页
- 了解实时时钟RTC的原理并通过stm32实现STM32的日历读取、设置和输出