K8S Node节点部署

  • 1、部署kubelet

（1）二进制包准备
[root@linux-node1 ~]# cd /usr/local/src/kubernetes/server/bin/[root@linux-node1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin/[root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.120:/opt/kubernetes/bin/[root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.130:/opt/kubernetes/bin/

（2）创建角色绑定

kubelet启动时会向kube-apiserver发送tsl bootstrap请求，所以需要将bootstrap的token设置成对应的角色，这样kubectl才有权限创建该请求。

[root@linux-node1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding"kubelet-bootstrap" created

（3）创建 kubelet bootstrapping kubeconfig 文件 设置集群参数
[root@linux-node1 ~]# cd /usr/local/src/ssl[root@linux-node1 ssl]# kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true\--server=https://192.168.56.110:6443 \--kubeconfig=bootstrap.kubeconfig
Cluster"kubernetes" set.

（4）设置客户端认证参数
[root@linux-node1 ssl]# kubectl config set-credentials kubelet-bootstrap \--token=ad6d5bb607a186796d8861557df0d17f \--kubeconfig=bootstrap.kubeconfig
User"kubelet-bootstrap" set.

（5）设置上下文参数
[root@linux-node1 ssl]# kubectl config set-context default \--cluster=kubernetes \--user=kubelet-bootstrap \--kubeconfig=bootstrap.kubeconfig
Context"default" created.

（6）选择默认上下文
[root@linux-node1 ssl]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
Switched to context"default".
[root@linux-node1 ssl]# cp bootstrap.kubeconfig /opt/kubernetes/cfg
[root@linux-node1 ssl]# scp bootstrap.kubeconfig 192.168.56.120:/opt/kubernetes/cfg
[root@linux-node1 ssl]# scp bootstrap.kubeconfig 192.168.56.130:/opt/kubernetes/cfg

  • 2、部署kubelet 1.设置CNI支持

（1）配置CNI
[root@linux-node2 ~]# mkdir -p /etc/cni/net.d
[root@linux-node2 ~]# vim /etc/cni/net.d/10-default.conf
{"name": "flannel","type": "flannel","delegate": {"bridge": "docker0","isDefaultGateway": true,"mtu": 1400}
}
[root@linux-node3 ~]# mkdir -p /etc/cni/net.d
[root@linux-node2 ~]# scp /etc/cni/net.d/10-default.conf 192.168.56.130:/etc/cni/net.d/10-default.conf

（2）创建kubelet数据存储目录
[root@linux-node2 ~]# mkdir /var/lib/kubelet
[root@linux-node3 ~]# mkdir /var/lib/kubelet

（3）创建kubelet服务配置

[root@linux-node2 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \--address=192.168.56.120\--hostname-override=192.168.56.120\--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0\--experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \--cert-dir=/opt/kubernetes/ssl \--network-plugin=cni \--cni-conf-dir=/etc/cni/net.d \--cni-bin-dir=/opt/kubernetes/bin/cni \--cluster-dns=10.1.0.2\--cluster-domain=cluster.local. \--hairpin-mode hairpin-veth \--allow-privileged=true\--fail-swap-on=false\--logtostderr=true\--v=2\--logtostderr=false\--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5[root@linux-node3 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \--address=192.168.56.130\--hostname-override=192.168.56.130\--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0\--experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \--cert-dir=/opt/kubernetes/ssl \--network-plugin=cni \--cni-conf-dir=/etc/cni/net.d \--cni-bin-dir=/opt/kubernetes/bin/cni \--cluster-dns=10.1.0.2\--cluster-domain=cluster.local. \--hairpin-mode hairpin-veth \--allow-privileged=true\--fail-swap-on=false\--logtostderr=true\--v=2\--logtostderr=false\--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5

View Code

（4）启动Kubelet
[root@linux-node2 ~]# systemctl daemon-reload
[root@linux-node2 ~]# systemctl enable kubelet
[root@linux-node2 ~]# systemctl start kubelet
[root@linux-node2 kubernetes]# systemctl status kubelet[root@linux-node3 ~]# systemctl daemon-reload
[root@linux-node3 ~]# systemctl enable kubelet
[root@linux-node3 ~]# systemctl start kubelet
[root@linux-node3 kubernetes]# systemctl status kubelet

在查看kubelet的状态，发现有如下报错Failed to get system container stats for "/system.slice/kubelet.service": failed to...此时需要调整kubelet的启动参数。

解决方法：在/usr/lib/systemd/system/kubelet.service的[service]新增:Environment="KUBELET_MY_ARGS=--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice"修改ExecStart: 在末尾新增$KUBELET_MY_ARGS

[root@linux-node2 system]# systemctl status kubelet
● kubelet.service-Kubernetes KubeletLoaded: loaded (/usr/lib/systemd/system/kubelet.service; static; vendor preset: disabled)Active: active (running) since 四2018-05-31 16:33:17CST; 16h agoDocs: https://github.com/GoogleCloudPlatform/kubernetesMain PID: 53223(kubelet)CGroup:/system.slice/kubelet.service└─53223 /opt/kubernetes/bin/kubelet --address=192.168.56.120 --hostname-override=192.168.56.120 --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 --experiment...6月01 08:51:09 linux-node2.example.com kubelet[53223]: E0601 08:51:09.355765   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:51:19 linux-node2.example.com kubelet[53223]: E0601 08:51:19.363906   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:51:29 linux-node2.example.com kubelet[53223]: E0601 08:51:29.385439   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:51:39 linux-node2.example.com kubelet[53223]: E0601 08:51:39.393790   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:51:49 linux-node2.example.com kubelet[53223]: E0601 08:51:49.401081   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:51:59 linux-node2.example.com kubelet[53223]: E0601 08:51:59.407863   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:52:09 linux-node2.example.com kubelet[53223]: E0601 08:52:09.415552   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:52:19 linux-node2.example.com kubelet[53223]: E0601 08:52:19.425998   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:52:29 linux-node2.example.com kubelet[53223]: E0601 08:52:29.443804   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
6月01 08:52:39 linux-node2.example.com kubelet[53223]: E0601 08:52:39.450814   53223 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to...
Hint: Some lines were ellipsized, use-l to show in full.

（5）查看csr请求 注意是在linux-node1上执行。
[root@linux-node1 ssl]# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-6Wc7kmqBIaPOw83l2F1uCKN-uUaxfkVhIU8K93S5y1U   1m        kubelet-bootstrap   Pending
node-csr-fIXcxO7jyR1Au7nrpUXht19eXHnX1HdFl99-oq2sRsA   1m        kubelet-bootstrap   Pending

（6）批准kubelet 的 TLS 证书请求
[root@linux-node1 ssl]#  kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargskubectl certificate approve
certificatesigningrequest.certificates.k8s.io"node-csr-6Wc7kmqBIaPOw83l2F1uCKN-uUaxfkVhIU8K93S5y1U"approved
certificatesigningrequest.certificates.k8s.io"node-csr-fIXcxO7jyR1Au7nrpUXht19eXHnX1HdFl99-oq2sRsA"approved[root@linux-node1 ssl]# kubectl get csr
NAME                                                   AGE       REQUESTOR           CONDITION
node-csr-6Wc7kmqBIaPOw83l2F1uCKN-uUaxfkVhIU8K93S5y1U   2m        kubelet-bootstrap   Approved,Issued
node-csr-fIXcxO7jyR1Au7nrpUXht19eXHnX1HdFl99-oq2sRsA   2m        kubelet-bootstrap   Approved,Issued执行完毕后，查看节点状态已经是Ready的状态了
[root@linux-node1 ssl]# kubectl get node
NAME             STATUS    ROLES     AGE       VERSION192.168.56.120   Ready     <none>    50m       v1.10.1
192.168.56.130   Ready     <none>    46m       v1.10.1

  • 3、部署Kubernetes Proxy

（1）配置kube-proxy使用LVS
[root@linux-node2 ~]# yum install -y ipvsadm ipset conntrack
[root@linux-node3 ~]# yum install -y ipvsadm ipset conntrack

（2）创建 kube-proxy 证书请求
[root@linux-node1 ~]# cd /usr/local/src/ssl/[root@linux-node1 ssl]# vim kube-proxy-csr.json
{"CN": "system:kube-proxy","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "BeiJing","L": "BeiJing","O": "k8s","OU": "System"}]
}

（3）生成证书
[root@linux-node1~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \-ca-key=/opt/kubernetes/ssl/ca-key.pem \-config=/opt/kubernetes/ssl/ca-config.json \-profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

（4）分发证书到所有Node节点
[root@linux-node1 ssl]# cp kube-proxy*.pem /opt/kubernetes/ssl/[root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.120:/opt/kubernetes/ssl/[root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.120:/opt/kubernetes/ssl/

（5）创建kube-proxy配置文件
[root@linux-node1 ssl]# kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true\--server=https://192.168.56.110:6443 \--kubeconfig=kube-proxy.kubeconfig
Cluster"kubernetes"set.[root@linux-node1 ssl]# kubectl config set-credentials kube-proxy \--client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \--client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \--embed-certs=true\--kubeconfig=kube-proxy.kubeconfig
User"kube-proxy"set.[root@linux-node1 ssl]# kubectl config set-context default \--cluster=kubernetes \--user=kube-proxy \--kubeconfig=kube-proxy.kubeconfig
Context"default"created.[root@linux-node1 ssl]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
Switched to context"default".

（6）分发kubeconfig配置文件
[root@linux-node1 ssl]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg/[root@linux-node1 ssl]# scp kube-proxy.kubeconfig 192.168.56.120:/opt/kubernetes/cfg/[root@linux-node1 ssl]# scp kube-proxy.kubeconfig 192.168.56.130:/opt/kubernetes/cfg/

（7）创建kube-proxy服务配置
[root@linux-node1 ssl]# mkdir /var/lib/kube-proxy
[root@linux-node2 ssl]# mkdir /var/lib/kube-proxy
[root@linux-node3 ssl]# mkdir /var/lib/kube-proxy[root@linux-node1 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \--bind-address=192.168.56.120\--hostname-override=192.168.56.120\--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \--masquerade-all \--feature-gates=SupportIPVSProxyMode=true\--proxy-mode=ipvs \--ipvs-min-sync-period=5s \--ipvs-sync-period=5s \--ipvs-scheduler=rr \--logtostderr=true\--v=2\--logtostderr=false\--log-dir=/opt/kubernetes/logRestart=on-failure
RestartSec=5LimitNOFILE=65536[Install]
WantedBy=multi-user.target[root@linux-node1 ssl]# scp /usr/lib/systemd/system/kube-proxy.service 192.168.56.120:/usr/lib/systemd/system/kube-proxy.service
kube-proxy.service                                         100%  701   109.4KB/s   00:00[root@linux-node1 ssl]# scp /usr/lib/systemd/system/kube-proxy.service 192.168.56.130:/usr/lib/systemd/system/kube-proxy.service
kube-proxy.service                                         100%  701    34.9KB/s   00:00

（8）启动Kubernetes Proxy
[root@linux-node2 ~]# systemctl daemon-reload
[root@linux-node2 ~]# systemctl enable kube-proxy
[root@linux-node2 ~]# systemctl start kube-proxy
[root@linux-node2 ~]# systemctl status kube-proxy[root@linux-node3 ~]# systemctl daemon-reload
[root@linux-node3 ~]# systemctl enable kube-proxy
[root@linux-node3 ~]# systemctl start kube-proxy
[root@linux-node3 ~]# systemctl status kube-proxy

检查LVS状态，可以看到已经创建了一个LVS集群，将来自10.1.0.1:443的请求转到192.168.56.110:6443，而6443就是api-server的端口
[root@linux-node2 ~]# ipvsadm -Ln
IP Virtual Server version1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags->RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP10.1.0.1:443 rr persistent 10800-> 192.168.56.110:6443          Masq    1      0          0[root@linux-node3 ~]# ipvsadm -Ln
IP Virtual Server version1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags->RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP10.1.0.1:443 rr persistent 10800-> 192.168.56.110:6443          Masq    1      0          0         如果你在两台实验机器都安装了kubelet和proxy服务，使用下面的命令可以检查状态：[root@linux-node1 ssl]#  kubectl get node
NAME            STATUS    ROLES     AGE       VERSION192.168.56.120   Ready     <none>    22m       v1.10.1
192.168.56.130   Ready     <none>    3m        v1.10.1

到此，K8S的集群就部署完毕，由于K8S本身不支持网络，需要借助第三方网络才能进行创建Pod，将在下一节学习Flannel网络为K8S提供网络支持。

（9）遇到的问题：kubelet无法启动，kubectl get node 提示：no resource found

[root@linux-node1 ssl]#  kubectl get node
No resources found.[root@linux-node3 ~]# systemctl status kubelet
● kubelet.service-Kubernetes KubeletLoaded: loaded (/usr/lib/systemd/system/kubelet.service; static; vendor preset: disabled)Active: activating (auto-restart) (Result: exit-code) since Wed 2018-05-30 04:48:29EDT; 1s agoDocs: https://github.com/GoogleCloudPlatform/kubernetesProcess: 16995 ExecStart=/opt/kubernetes/bin/kubelet --address=192.168.56.130 --hostname-override=192.168.56.130 --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --cert-dir=/opt/kubernetes/ssl --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/kubernetes/bin/cni --cluster-dns=10.1.0.2 --cluster-domain=cluster.local. --hairpin-mode hairpin-veth --allow-privileged=true --fail-swap-on=false --logtostderr=true --v=2 --logtostderr=false --log-dir=/opt/kubernetes/log (code=exited, status=255)Main PID:16995 (code=exited, status=255)May30 04:48:29 linux-node3.example.com systemd[1]: Unit kubelet.service entered failed state.
May30 04:48:29 linux-node3.example.com systemd[1]: kubelet.service failed.
[root@linux-node3 ~]# tailf /var/log/messages
......
May30 04:46:24 linux-node3 kubelet: F0530 04:46:24.134612   16207 server.go:233] failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"提示kubelet使用的cgroup驱动类型和docker的cgroup驱动类型不一致。进行查看docker.service[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target
Wants=docker-storage-setup.service
Requires=docker-cleanup.timer[Service]
Type=notify
NotifyAccess=all
KillMode=process
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
EnvironmentFile=-/etc/sysconfig/docker-network
Environment=GOTRACEBACK=crash
Environment=DOCKER_HTTP_HOST_COMPAT=1Environment=PATH=/usr/libexec/docker:/usr/bin:/usr/sbin
ExecStart=/usr/bin/dockerd-current \--add-runtime docker-runc=/usr/libexec/docker/docker-runc-current \--default-runtime=docker-runc \--exec-opt native.cgroupdriver=systemd \   ###修改此处"systemd"为"cgroupfs"--userland-proxy-path=/usr/libexec/docker/docker-proxy-current \$OPTIONS \$DOCKER_STORAGE_OPTIONS \$DOCKER_NETWORK_OPTIONS \$ADD_REGISTRY \$BLOCK_REGISTRY \$INSECURE_REGISTRY
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576LimitNPROC=1048576LimitCORE=infinity
TimeoutStartSec=0Restart=on-abnormal
MountFlags=slave[Install]
WantedBy=multi-user.target
[root@linux-node3 ~]# systemctl daemon-reload
[root@linux-node3 ~]# systemctl restart docker.service
[root@linux-node3 ~]# systemctl restart kubelet

转载于:https://www.cnblogs.com/linuxk/p/9272778.html

Kubernetes学习之路（四）之Node节点二进制部署相关推荐

  1. Kubernetes学习之路目录

    Kubernetes基础篇 环境说明 版本说明 系统环境 Centos 7.2 Kubernetes版本 v1.11.2 Docker版本 v18.09 Kubernetes学习之路(一)之概念和架构 ...

  2. JavaScript学习（十四）—元素节点关系和特殊节点

    JavaScript学习(十四)-元素节点关系和特殊节点 一.元素节点 (1).parentElement: 获取某元素的父元素,它和parentNode的区别是parentElement获取到的值时 ...

  3. Java多线程学习之路(四)---死锁(DeadLock)

    Java多线程学习之路(四)-死锁(DeadLock) 1.定义 死锁就是多个线程在竞争共享资源的时候,相互阻塞,不能脱身的状态(个人理解).其实死锁一定程度上可以看成一个死循环. 举个现实生活中的例 ...

  4. typescript学习之路(四) —— ts类的继承（包含es5以及es6的类继承）

    上一文已经写了es5,es6等类的定义,所以本章主要写es5和es6的继承,由于es6的继承和ts的继承如出一辙,只是加了类型定义而已,所以ts的继承稍微写下,不会太详细. 文章目录 es5继承 原型 ...

  5. 认识kubernetes（k8s），k8s单节点etcd部署

    认识kubernetes(k8s),k8s单节点etcd部署 一.k8s概述 (一).k8s简介 (二).k8s特性 (三).k8s群集架构与组件 (四).k8s核心概念 (五).k8s三种部署方式 ...

  6. Kubernetes多节点二进制部署

    Kubernetes多节点二进制部署 一.部署master02 节点 修改主机名,关闭防火墙 在k8smaster01上操作 在k8smaster02上操作 二.部署负载均衡 1.配置nginx的官方 ...

  7. K8S——单master节点和基于单master节点的双master节点二进制部署（本机实验，防止卡顿，所以多master就不做3台了）

    K8S--单master节点和基于单master节点的双master节点二进制部署 一.准备 二.ETCD集群 1.master节点 2.node节点 三.Flannel网络部署 四.测试容器间互通 ...

  8. Linux运维容器篇 k8s单节点二进制部署（1） ECTD部署+CA证书制作

    文章目录 一.环境配置 二.制作CA证书 1.传入脚本 2.创建CA证书 3 指定节点通讯证书 三 部署etcd集群 1 安装etcd包并传入证书 配置etcd启动脚本并生成cfg文件 配置node节 ...

  9. Kubernetes学习之路（26）之kubeasz+ansible部署集群

    目录 1.环境说明 2.准备工作 3.分步骤安装 3.1.创建证书和安装准备 3.2.安装etcd集群 3.3.安装docker 3.4.安装master节点 3.5.安装node节点 3.6.部署集 ...

最新文章

  1. 用Eclipse进行远程Debug代码
  2. 自学python系列10：python的函数和函数式编程
  3. c++ primer学习笔记(2)-c++基本数据类型
  4. MySQL数据库InnoDB存储引擎中的锁机制--转载
  5. 华为杯数学建模优秀论文_数学建模经典例题（2011年国赛A题与优秀论文）
  6. android加法服务类,iOS越来越像Android:苹果简单做加法远离精致
  7. linux下提示libpng12-0缺失
  8. Java中截取字符串中小数点前面的字符
  9. mysql提高运行效率_提升Mysql执行效率的SQL优化技巧汇总
  10. RPM包安装卸载命令
  11. 江苏省考计算机类包括哪些专业,2019年江苏省公务员考试计算机类包括哪些专业..._公务员考试_帮考网...
  12. 在JavaScript中NaN为什么不等于NaN
  13. 使用eclipse时出现cannot access compilation unit的解决方法
  14. 百度在线语音合成API接口简单应用
  15. 一种内嵌P2P的wifi转红外发射神器
  16. 浮生若梦，静如止水，不问情意，只愿你安好
  17. c语言高级程序知识,《高级语言程序设计》知识点总结（一）
  18. 入门级选手浅写一下关于前端的知识点
  19. e1000网卡驱动第二天
  20. YOLOv3中的非极大值抑制

热门文章

  1. 利用JQuery jsonp实现Ajax跨域请求 .Net 的*.handler 和 WebService，返回json数据
  2. python3（十四）Python 异常处理
  3. python 数据科学书籍_您必须在2020年阅读的数据科学书籍
  4. 央行允许银行倒闭破产，那么储户的存款怎么办？
  5. spring boot mybatis 整合_MyBatis学习：MyBatis和Spring整合
  6. resnet keras 结构_Day146：第二讲 ResNet
  7. visual studio 设计器不显示_与城共生：南京朝天宫“参与性”城市设计
  8. 苹果6发布时间_苹果秋季发布会将在北京时间9月16日举办
  9. matlab内存溢出的解决方案
  10. 图像相似性搜索的原理