微软OpenPai平台部署安装(kubernetes 大数据和深度学习平台安装部署)
（1）特别注意，机器配置必须16/32核 64G内存，曾经在虚拟机上安装部署8核/28G内存走不出来。哪怕修改pod文件memory配置
（2）另外遇到镜像下载不出来建议先下载到本地
（3）操作系统ubuntu16.04 cpu，不用GPU也可以
我采用的华为云部署，每个小时大概7-8元
参考网址
https://hub.docker.com/
https://gitee.com/zhangjinyu/pai.git 提高下载速度
root@ubuntu:~# cat huawei1.sh
配置阿里源
cat /dev/null > /etc/apt/sources.list
tee /etc/apt/sources.list << 'EOF'
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted
deb http://mirrors.aliyun.com/ubuntu/ bionic universe
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates universe
deb http://mirrors.aliyun.com/ubuntu/ bionic multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted
deb http://mirrors.aliyun.com/ubuntu/ bionic-security universe
deb http://mirrors.aliyun.com/ubuntu/ bionic-security multiverse
EOF
跟新安装docker
apt-get update
apt-get install docker.io ssh ntp -y
配置docker阿里源，如果在华为云建立自己的镜像仓库，此处不需要加载
tee /etc/docker/daemon.json << 'EOF'
{
"registry-mirrors": ["https://hny2a055.mirror.aliyuncs.com"]
}
EOF
安装必要的软件包，与官网不一样有版本更新manpages-pl
apt-get -y install nano vim joe wget curl jq gawk psmisc python python-yaml python-jinja2 python-urllib3 python-tz python-nose python-prettytable python-netifaces python-dev python-pip python-mysqldb openjdk-8-jre openjdk-8-jdk openssh-server openssh-client git bash-completion inotify-tools rsync manpages-pl net-tools
英伟达驱动，目前cpu 此处也需要执行
curl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list
配置阿里云kubernetes源
apt-get update && apt-get install -y apt-transport-https
tee /etc/systemd/system/docker.service.d/kolla.conf << 'EOF'
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
更新安装python软件包采用阿里源
apt-get update
pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install --upgrade urllib3 -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install --upgrade ipaddress -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install --upgrade python-etcd docker kubernetes paramiko==2.6.0 GitPython -i https://pypi.tuna.tsinghua.edu.cn/simple
pip install --ignore-installed PyYAML -i https://pypi.tuna.tsinghua.edu.cn/simple
python -m easy_install --upgrade pyOpenSSL
下载华为云我的镜像， 可以提前下载下来，采用docker pull openpai/* 或者不需要执行这一步骤，我主要为了加速
docker login -u cn-north-1@FFHXH5GE4NWNM7EGVUZK -p 9b82020ba72dd32e8efe68ccd9d513b3069e6362dd744b9fa250856cf0908c81 swr.cn-north-1.myhuaweicloud.com
#dev-box
FILE="
kube-runtime
webportal
watchdog
yarn-frameworklauncher
end-to-end-test
zookeeper
cleaning-image
hadoop-run
internal-storage-create
base-image
postgresql
job-exporter
yarn-exporter
etcd-upgrade
drivers-384.111
drivers-390.25
drivers-410.73
drivers-418.56
pylon
rest-server
fluentd
cleaner
storage-manager
grafana
log-manager-logrotate
log-manager-nginx
node-exporter
"
for i in $FILE
do
echo $i
#echo docker tag $i swr.cn-east-2.myhuaweicloud.com/xsjzdrxsjzdr/$i
docker pull swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/$i
#docker rmi openpai/$i
docker tag swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/$i openpai/$i
#docker pull openpai/$i
done
root@ubuntu:~# cat ha | grep openpai
openpai/job-exporter latest 460254b6072a 14 months ago 1.06GB
openpai/watchdog latest 0f82b772a9d9 14 months ago 992MB
openpai/rest-server latest 47f6bf04be6c 14 months ago 1.04GB
openpai/webportal latest 3c501b3a40d1 14 months ago 1.51GB
openpai/yarn-frameworklauncher latest 591fc12748ce 15 months ago 1.93GB
openpai/end-to-end-test latest 4fd69f739b5e 15 months ago 2.1GB
openpai/hadoop-run latest d596e208c837 15 months ago 1.74GB
openpai/cleaning-image latest 806775ef0540 15 months ago 967MB
openpai/zookeeper latest 4de8da83bded 15 months ago 917MB
openpai/base-image latest 7fe7fc9abc00 15 months ago 874MB
openpai/cleaner latest 2fc855208a24 15 months ago 1.03GB
openpai/yarn-exporter latest 8d0e652c4e8a 15 months ago 942MB
openpai/grafana latest 0e8a285d92b7 15 months ago 283MB
openpai/drivers-384.111 latest 02908196dcc3 15 months ago 3.07GB
openpai/drivers-390.25 latest 3ad4a1f2f46b 15 months ago 3.76GB
openpai/drivers-410.73 latest 836163c091ef 15 months ago 3.88GB
openpai/pylon latest 4510b0b6c8f2 15 months ago 520MB
openpai/hyperkube v1.9.9 06696cbe4586 22 months ago 652MB
openpai/kube-controller-manager v1.9.9 2f4813f0ccc3 22 months ago 140MB
openpai/kube-apiserver v1.9.9 09a56741e8c8 22 months ago 213MB
openpai/kube-scheduler v1.9.9 74cea6a9ab4d 22 months ago 63.7MB
openpai/node-exporter latest 188af75e2de0 24 months ago 22.9MB
openpai/etcd 3.2.17 0d36cbbcc544 2 years ago 229MB
openpai/kubernetes-dashboard-amd64 v1.8.3 0c60bcf89900 2 years ago 102MB
openpai/prometheus latest c8ecf7c719c1 2 years ago 112MB
openpai/pause-amd64 3.0 99e59f495ffa 4 years ago 747kB
搭建本地镜像仓库
启动镜像
docker run -itd -e COLUMNS=$COLUMNS -e LINES=$LINES -e TERM=$TERM -v /var/lib/docker:/var/lib/docker -v /var/run/docker.sock:/var/run/docker.sock -v /pathHadoop:/pathHadoop -v /pathConfiguration:/cluster-configuration --pid=host --privileged=true --net=host --name=dev-box docker.io/openpai/dev-box
docker exec -it dev-box /bin/bash
vi /pai/deployment/quick-start/quick-start.yaml
machines:
- 192.168.229.240 #虚拟机的IP地址，第一个为master节点，其它为slave节点
ssh-username: root #虚拟机root账号
ssh-password: hadoop #虚拟机root密码
在此我没有采用容器openpai/dev-box启动部署，而是把dev-box 容器目录中的/pai软件包拷贝到物理机器中的/pai目录中
scp 虚拟机到物理机器中
/pai
/root/paiconfig
root@ubuntu:/pai# python paictl.py config generate -i /pai/deployment/quick-start/quick-start.yaml -o ~/pai-config -f
root@hadoop:~/pai-config# vi kubernetes-configuration.yaml
31 docker-registry: docker.io/openpai docker-registry: swr.cn-east-2.myhuaweicloud.com/xsjzdrxsjzdr
没有GPU则不需要加
root@hadoop:~/pai-config# vi services-configuration.yaml
20 cluster:
21 #common:
22 # cluster-id: pai-example
23 # cluster-type: k8s
24 # # HDFS, zookeeper data path on your cluster machine.
25
26 nvidia-drivers-version:384.111
root@hadoop:/pai# ./paictl.py cluster k8s-bootup -p ~/pai-config
root@hadoop:/pai# python paictl.py config get-id #如果没有输出，可以随意填写在下面的push命令中
root@hadoop:/pai# python paictl.py config push -p ~/pai-config
cluster ID: fc7aeaf3c09455366720924638477af1201c730176b4201eb03a7e31ca1650a9
启动openpai服务
python paictl.py service start
cluster ID: fc7aeaf3c09455366720924638477af1201c730176b4201eb03a7e31ca1650a9
停掉大数据集群
python paictl.py service start
清理环境
root@hadoop:/pai# ./paictl.py cluster k8s-clean -p ~/pai-config
安装日志
root@ecs-45ae:/pai# python paictl.py config push -p ~/pai-config
/pai/deployment/paiLibrary/common/file_handler.py:37: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
cluster_data = yaml.load(f)
2020-05-10 21:17:32,860 [INFO] - deployment.confStorage.synchronization : The temporary cluster configuration path is : ./tmp-config-1589116652
Please input the cluster-id which you wanna operate: fc7aeaf3c09455366720924638477af1201c730176b4201eb03a7e31ca1650a9
2020-05-10 21:17:48,558 [INFO] - deployment.confStorage.upload : Congratulations: Cluster-id checking passed.
2020-05-10 21:17:48,570 [INFO] - deployment.paiLibrary.common.kubernetes_handler : configmap named pai-configuration is updated.
2020-05-10 21:17:48,570 [INFO] - deployment.confStorage.synchronization : Cluster Configuration synchronization from external storage is successful.
root@ecs-45ae:/pai# python paictl.py service start
/pai/deployment/paiLibrary/common/file_handler.py:37: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
cluster_data = yaml.load(f)
2020-05-10 21:18:00,999 [INFO] - deployment.paiLibrary.paiService.service_management_start : Get the service-list to manage : ['hadoop-data-node', 'hadoop-name-node', 'drivers', 'hadoop-node-manager', 'webportal', 'cleaner', 'rest-server', 'hadoop-resource-manager', 'grafana', 'end-to-end-test', 'watchdog', 'alert-manager', 'node-exporter', 'yarn-frameworklauncher', 'prometheus', 'zookeeper', 'hadoop-batch-job', 'cluster-configuration', 'job-exporter', 'hadoop-jobhistory', 'pylon']
Please input the cluster-id which you wanna operate: fc7aeaf3c09455366720924638477af1201c730176b4201eb03a7e31ca1650a9
2020-05-10 21:18:04,102 [INFO] - deployment.confStorage.download : Congratulations: Cluster-id checking passed.
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation for each service parser.
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of rest-server
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of rest-server is passed
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of hadoop-data-node
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of hadoop-data-node is passed
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of hadoop-jobhistory
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of hadoop-jobhistory is passed
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of yarn-frameworklauncher
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of yarn-frameworklauncher is passed
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of webportal
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of webportal is passed
2020-05-10 21:18:04,139 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of hadoop-name-node
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of hadoop-name-node is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of cleaner
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of cleaner is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of kubernetes
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of kubernetes is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of drivers
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of drivers is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of zookeeper
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of zookeeper is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of hadoop-batch-job
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of hadoop-batch-job is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of grafana
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of grafana is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of machine
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of machine is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of cluster
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of cluster is passed
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of prometheus
2020-05-10 21:18:04,140 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of prometheus is passed
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of alert-manager
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of alert-manager is passed
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of job-exporter
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of job-exporter is passed
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of pylon
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of pylon is passed
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of hadoop-node-manager
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of hadoop-node-manager is passed
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of node-exporter
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of node-exporter is passed
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do pre-validation of hadoop-resource-manager
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation of hadoop-resource-manager is passed
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Pre-validation is successful!
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate cluster object model.
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of rest-server.
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of rest-server is generated.
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of hadoop-data-node.
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of hadoop-data-node is generated.
2020-05-10 21:18:04,141 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of hadoop-jobhistory.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of hadoop-jobhistory is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of yarn-frameworklauncher.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of yarn-frameworklauncher is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of webportal.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of webportal is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of hadoop-name-node.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of hadoop-name-node is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of cleaner.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of cleaner is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of kubernetes.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of kubernetes is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of drivers.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of drivers is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of zookeeper.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of zookeeper is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of hadoop-batch-job.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of hadoop-batch-job is generated.
2020-05-10 21:18:04,142 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of grafana.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of grafana is generated.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of machine.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of machine is generated.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of cluster.
2020-05-10 21:18:04,143 [INFO] - cluster : docker registry authentication not provided
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of cluster is generated.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of prometheus.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of prometheus is generated.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of alert-manager.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of alert-manager is generated.
2020-05-10 21:18:04,143 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of job-exporter.
2020-05-10 21:18:04,144 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of job-exporter is generated.
2020-05-10 21:18:04,144 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of pylon.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of pylon is generated.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of hadoop-node-manager.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of hadoop-node-manager is generated.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of node-exporter.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of node-exporter is generated.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do generate object model of hadoop-resource-manager.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Object model of hadoop-resource-manager is generated.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Cluster Object Model is generated.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of rest-server.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of rest-server is passed.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of hadoop-data-node.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of hadoop-data-node is passed.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of hadoop-jobhistory.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of hadoop-jobhistory is passed.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of yarn-frameworklauncher.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of yarn-frameworklauncher is passed.
2020-05-10 21:18:04,145 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of webportal.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of webportal is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of hadoop-name-node.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of hadoop-name-node is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of cleaner.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of cleaner is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of kubernetes.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of kubernetes is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of drivers.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of drivers is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of zookeeper.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of zookeeper is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of hadoop-batch-job.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of hadoop-batch-job is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of grafana.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of grafana is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of machine.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of machine is passed.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of cluster.
2020-05-10 21:18:04,146 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of cluster is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of prometheus.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of prometheus is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of alert-manager.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of alert-manager is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of job-exporter.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of job-exporter is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of pylon.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of pylon is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of hadoop-node-manager.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of hadoop-node-manager is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of node-exporter.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of node-exporter is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Begin to do post-validation of hadoop-resource-manager.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation of hadoop-resource-manager is passed.
2020-05-10 21:18:04,147 [INFO] - deployment.clusterObjectModel.cluster_object_model : Post-validation is successful!
2020-05-10 21:18:04,151 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:18:04,152 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service cluster-configuration's template file
2020-05-10 21:18:04,152 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service cluster-configuration's configuration.
2020-05-10 21:18:04,152 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service cluster-configuration.
2020-05-10 21:18:04,152 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service cluster-configuration is created.
2020-05-10 21:18:04,152 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/secret.yaml.template.
2020-05-10 21:18:04,152 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/secret.yaml.
2020-05-10 21:18:04,155 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/host-configuration/host-configuration.yaml.template.
2020-05-10 21:18:04,155 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/host-configuration/host-configuration.yaml.
2020-05-10 21:18:04,161 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/gpu-configuration/gpu-configuration.yml.template.
2020-05-10 21:18:04,161 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/gpu-configuration/gpu-configuration.yml.
2020-05-10 21:18:04,164 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/gpu-configuration/gpu-configuration.json.template.
2020-05-10 21:18:04,164 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/gpu-configuration/gpu-configuration.json.
2020-05-10 21:18:04,169 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/docker-credentials/config.json.template.
2020-05-10 21:18:04,169 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/docker-credentials/config.json.
2020-05-10 21:18:04,169 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/stop.sh.template.
2020-05-10 21:18:04,170 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/stop.sh.
2020-05-10 21:18:04,171 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/start.sh.template.
2020-05-10 21:18:04,171 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cluster-configuration/deploy/start.sh.
2020-05-10 21:18:04,176 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service cluster-configuration is generated.
2020-05-10 21:18:04,176 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ cluster-configuration ]
2020-05-10 21:18:04,176 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service cluster-configuration's start script.
W0510 21:18:04.223283 5785 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/host-configuration created
W0510 21:18:04.340644 5830 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/docker-credentials created
W0510 21:18:04.445448 5867 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/gpu-configuration created
secret/pai-secret created
node/192.168.114.178 labeled
node/192.168.114.178 labeled
2020-05-10 21:18:04,722 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:18:04,723 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of cluster-configuration's service.
2020-05-10 21:18:04,723 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of cluster-configuration's service have been cleaned up.
2020-05-10 21:18:04,723 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start cluster-configuration
2020-05-10 21:18:04,723 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:18:04,725 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:18:04,726 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service drivers's template file
2020-05-10 21:18:04,726 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service drivers's configuration.
2020-05-10 21:18:04,726 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service drivers.
2020-05-10 21:18:04,726 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service drivers is created.
2020-05-10 21:18:04,726 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/drivers/deploy/drivers.yaml.template.
2020-05-10 21:18:04,726 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/drivers/deploy/drivers.yaml.
/pai/deployment/paiLibrary/paiService/service_template_generate.py:70: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(str_src_yaml)
2020-05-10 21:18:04,746 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/drivers/deploy/clean.yaml.template.
2020-05-10 21:18:04,746 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/drivers/deploy/clean.yaml.
2020-05-10 21:18:04,756 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/drivers/deploy/delete.sh.template.
2020-05-10 21:18:04,756 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/drivers/deploy/delete.sh.
2020-05-10 21:18:04,757 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service drivers is generated.
2020-05-10 21:18:04,758 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ drivers ]
2020-05-10 21:18:04,758 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service drivers's start script.
daemonset.apps/drivers-one-shot created
drivers-one-shot is not ready yet. Please wait for a moment!
drivers-one-shot is not ready yet. Please wait for a moment!
drivers-one-shot is ready!
2020-05-10 21:18:35,200 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:18:35,200 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of drivers's service.
2020-05-10 21:18:35,200 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of drivers's service have been cleaned up.
2020-05-10 21:18:35,200 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start drivers
2020-05-10 21:18:35,201 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:18:35,203 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:18:35,203 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service zookeeper's template file
2020-05-10 21:18:35,203 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service zookeeper's configuration.
2020-05-10 21:18:35,203 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service zookeeper.
2020-05-10 21:18:35,203 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service zookeeper is created.
2020-05-10 21:18:35,203 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/zookeeper/deploy/zookeeper.yaml.template.
2020-05-10 21:18:35,203 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/zookeeper/deploy/zookeeper.yaml.
2020-05-10 21:18:35,218 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/zookeeper/deploy/delete.yaml.template.
2020-05-10 21:18:35,218 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/zookeeper/deploy/delete.yaml.
2020-05-10 21:18:35,220 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service zookeeper is generated.
2020-05-10 21:18:35,221 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ zookeeper ]
2020-05-10 21:18:35,221 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service zookeeper's start script.
W0510 21:18:35.268990 6389 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/zk-configuration created
daemonset.apps/zookeeper-ds created
zookeeper is not ready yet. Please wait for a moment!
zookeeper is ready!
2020-05-10 21:18:55,761 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:18:55,761 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of zookeeper's service.
2020-05-10 21:18:55,761 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of zookeeper's service have been cleaned up.
2020-05-10 21:18:55,761 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start zookeeper
2020-05-10 21:18:55,761 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:18:55,764 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:18:55,764 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service hadoop-name-node's template file
2020-05-10 21:18:55,765 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service hadoop-name-node's configuration.
2020-05-10 21:18:55,765 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service hadoop-name-node.
2020-05-10 21:18:55,765 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service hadoop-name-node is created.
2020-05-10 21:18:55,765 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-name-node/deploy/hadoop-name-node.yaml.template.
2020-05-10 21:18:55,765 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-name-node/deploy/hadoop-name-node.yaml.
2020-05-10 21:18:55,780 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-name-node/deploy/delete.yaml.template.
2020-05-10 21:18:55,781 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-name-node/deploy/delete.yaml.
2020-05-10 21:18:55,783 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service hadoop-name-node is generated.
2020-05-10 21:18:55,783 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ hadoop-name-node ]
2020-05-10 21:18:55,783 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service hadoop-name-node's start script.
W0510 21:18:55.828920 6885 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-name-node-configuration created
daemonset.apps/hadoop-name-node-ds created
hadoop-name-node is not ready yet. Please wait for a moment!
hadoop-name-node is not ready yet. Please wait for a moment!
hadoop-name-node is ready!
2020-05-10 21:19:26,951 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of hadoop-name-node's service.
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of hadoop-name-node's service have been cleaned up.
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start hadoop-name-node
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service hadoop-data-node's template file
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service hadoop-data-node's configuration.
2020-05-10 21:19:26,952 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service hadoop-data-node.
2020-05-10 21:19:26,953 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service hadoop-data-node is created.
2020-05-10 21:19:26,953 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-data-node/deploy/hadoop-data-node.yaml.template.
2020-05-10 21:19:26,953 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-data-node/deploy/hadoop-data-node.yaml.
2020-05-10 21:19:26,975 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-data-node/deploy/delete.yaml.template.
2020-05-10 21:19:26,975 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-data-node/deploy/delete.yaml.
2020-05-10 21:19:26,977 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service hadoop-data-node is generated.
2020-05-10 21:19:26,977 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ hadoop-data-node ]
2020-05-10 21:19:26,977 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service hadoop-data-node's start script.
W0510 21:19:27.023539 8076 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-data-node-configuration created
daemonset.apps/hadoop-data-node-ds created
hadoop-data-node is not ready yet. Please wait for a moment!
hadoop-data-node is ready!
2020-05-10 21:19:47,523 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:19:47,524 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of hadoop-data-node's service.
2020-05-10 21:19:47,524 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of hadoop-data-node's service have been cleaned up.
2020-05-10 21:19:47,524 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start hadoop-data-node
2020-05-10 21:19:47,524 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:19:47,529 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:19:47,529 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service hadoop-resource-manager's template file
2020-05-10 21:19:47,529 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service hadoop-resource-manager's configuration.
2020-05-10 21:19:47,530 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service hadoop-resource-manager.
2020-05-10 21:19:47,530 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service hadoop-resource-manager is created.
2020-05-10 21:19:47,530 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-resource-manager/deploy/hadoop-resource-manager.yaml.template.
2020-05-10 21:19:47,530 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-resource-manager/deploy/hadoop-resource-manager.yaml.
2020-05-10 21:19:47,555 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-resource-manager/deploy/delete.yaml.template.
2020-05-10 21:19:47,555 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-resource-manager/deploy/delete.yaml.
2020-05-10 21:19:47,558 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-resource-manager/deploy/hadoop-resource-manager-configuration/capacity-scheduler.xml.template.
2020-05-10 21:19:47,558 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-resource-manager/deploy/hadoop-resource-manager-configuration/capacity-scheduler.xml.
2020-05-10 21:19:47,562 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service hadoop-resource-manager is generated.
2020-05-10 21:19:47,562 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ hadoop-resource-manager ]
2020-05-10 21:19:47,562 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service hadoop-resource-manager's start script.
W0510 21:19:47.606606 9025 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-resource-manager-configuration created
daemonset.apps/hadoop-resource-manager-ds created
hadoop-resource-manager is not ready yet. Please wait for a moment!
hadoop-resource-manager is not ready yet. Please wait for a moment!
hadoop-resource-manager is not ready yet. Please wait for a moment!
hadoop-resource-manager is ready!
2020-05-10 21:20:28,126 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of hadoop-resource-manager's service.
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of hadoop-resource-manager's service have been cleaned up.
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start hadoop-resource-manager
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service hadoop-node-manager's template file
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service hadoop-node-manager's configuration.
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service hadoop-node-manager.
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service hadoop-node-manager is created.
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-node-manager/deploy/hadoop-node-manager.yaml.template.
2020-05-10 21:20:28,127 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-node-manager/deploy/hadoop-node-manager.yaml.
2020-05-10 21:20:28,162 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-node-manager/deploy/delete.yaml.template.
2020-05-10 21:20:28,162 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-node-manager/deploy/delete.yaml.
2020-05-10 21:20:28,166 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service hadoop-node-manager is generated.
2020-05-10 21:20:28,166 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ hadoop-node-manager ]
2020-05-10 21:20:28,166 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service hadoop-node-manager's start script.
W0510 21:20:28.210821 11136 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-node-manager-configuration created
daemonset.apps/hadoop-node-manager-ds created
hadoop-node-manager is not ready yet. Please wait for a moment!
hadoop-node-manager is not ready yet. Please wait for a moment!
hadoop-node-manager is ready!
2020-05-10 21:20:58,743 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:20:58,744 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of hadoop-node-manager's service.
2020-05-10 21:20:58,744 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of hadoop-node-manager's service have been cleaned up.
2020-05-10 21:20:58,744 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start hadoop-node-manager
2020-05-10 21:20:58,744 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:20:58,755 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:20:58,755 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service hadoop-jobhistory's template file
2020-05-10 21:20:58,756 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service hadoop-jobhistory's configuration.
2020-05-10 21:20:58,756 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service hadoop-jobhistory.
2020-05-10 21:20:58,756 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service hadoop-jobhistory is created.
2020-05-10 21:20:58,756 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-jobhistory/deploy/hadoop-jobhistory.yaml.template.
2020-05-10 21:20:58,756 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-jobhistory/deploy/hadoop-jobhistory.yaml.
2020-05-10 21:20:58,778 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-jobhistory/deploy/delete.yaml.template.
2020-05-10 21:20:58,778 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-jobhistory/deploy/delete.yaml.
2020-05-10 21:20:58,781 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service hadoop-jobhistory is generated.
2020-05-10 21:20:58,781 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ hadoop-jobhistory ]
2020-05-10 21:20:58,781 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service hadoop-jobhistory's start script.
W0510 21:20:58.828257 13014 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-jobhistory-configuration created
daemonset.apps/hadoop-jobhistory-service created
hadoop-jobhistory-service is not ready yet. Please wait for a moment!
hadoop-jobhistory-service is not ready yet. Please wait for a moment!
hadoop-jobhistory-service is ready!
2020-05-10 21:21:29,341 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of hadoop-jobhistory's service.
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of hadoop-jobhistory's service have been cleaned up.
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start hadoop-jobhistory
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service hadoop-batch-job's template file
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service hadoop-batch-job's configuration.
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service hadoop-batch-job.
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service hadoop-batch-job is created.
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/hadoop-batch-job/deploy/one-time-job-hadoop.yaml.template.
2020-05-10 21:21:29,342 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/hadoop-batch-job/deploy/one-time-job-hadoop.yaml.
2020-05-10 21:21:29,346 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service hadoop-batch-job is generated.
2020-05-10 21:21:29,346 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ hadoop-batch-job ]
2020-05-10 21:21:29,346 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service hadoop-batch-job's start script.
W0510 21:21:29.391448 14953 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/hadoop-configuration created
No resources found in default namespace.
job.batch/batch-job-hadoop created
2020-05-10 21:21:39,633 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:21:39,633 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of hadoop-batch-job's service.
2020-05-10 21:21:39,633 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of hadoop-batch-job's service have been cleaned up.
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start hadoop-batch-job
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service yarn-frameworklauncher's template file
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service yarn-frameworklauncher's configuration.
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service yarn-frameworklauncher.
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service yarn-frameworklauncher is created.
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/yarn-frameworklauncher/deploy/yarn-frameworklauncher.yaml.template.
2020-05-10 21:21:39,634 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/yarn-frameworklauncher/deploy/yarn-frameworklauncher.yaml.
2020-05-10 21:21:39,655 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/yarn-frameworklauncher/deploy/delete.yaml.template.
2020-05-10 21:21:39,655 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/yarn-frameworklauncher/deploy/delete.yaml.
2020-05-10 21:21:39,657 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service yarn-frameworklauncher is generated.
2020-05-10 21:21:39,657 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ yarn-frameworklauncher ]
2020-05-10 21:21:39,658 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service yarn-frameworklauncher's start script.
W0510 21:21:39.702663 15633 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/yarn-frameworklauncher-configmap created
W0510 21:21:39.817797 15699 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/yarn-frameworklauncher-delete created
daemonset.apps/yarn-frameworklauncher-ds created
yarn-frameworklauncher is ready!
2020-05-10 21:21:50,287 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of yarn-frameworklauncher's service.
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of yarn-frameworklauncher's service have been cleaned up.
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start yarn-frameworklauncher
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service rest-server's template file
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service rest-server's configuration.
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service rest-server.
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service rest-server is created.
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/rest-server/deploy/rest-server.yaml.template.
2020-05-10 21:21:50,288 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/rest-server/deploy/rest-server.yaml.
2020-05-10 21:21:50,307 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/rest-server/deploy/start.sh.template.
2020-05-10 21:21:50,307 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/rest-server/deploy/start.sh.
2020-05-10 21:21:50,309 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service rest-server is generated.
2020-05-10 21:21:50,309 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ rest-server ]
2020-05-10 21:21:50,309 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service rest-server's start script.
2020-05-10 21:21:50,373 [INFO] - legacy_user_migrate.py:164 : Starts to migrate legacy user data from etcd to kubernetes secrets
2020-05-10 21:21:50,376 [INFO] - legacy_user_migrate.py:50 : No legacy user data found in etcd
2020-05-10 21:21:50,376 [INFO] - legacy_user_migrate.py:179 : No legacy data found
2020-05-10 21:21:50,376 [INFO] - legacy_user_migrate.py:183 : Legacy user data transfer from etcd to kubernetes secret successfully
daemonset.apps/rest-server-ds created
rest-server is not ready yet. Please wait for a moment!
rest-server is ready!
2020-05-10 21:22:10,810 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:22:10,810 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of rest-server's service.
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of rest-server's service have been cleaned up.
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start rest-server
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service webportal's template file
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service webportal's configuration.
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service webportal.
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service webportal is created.
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/webportal/deploy/webportal.yaml.template.
2020-05-10 21:22:10,811 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/webportal/deploy/webportal.yaml.
2020-05-10 21:22:10,827 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service webportal is generated.
2020-05-10 21:22:10,827 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ webportal ]
2020-05-10 21:22:10,828 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service webportal's start script.
daemonset.apps/webportal-ds created
webportal is not ready yet. Please wait for a moment!
webportal is ready!
2020-05-10 21:22:31,270 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:22:31,271 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of webportal's service.
2020-05-10 21:22:31,271 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of webportal's service have been cleaned up.
2020-05-10 21:22:31,271 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start webportal
2020-05-10 21:22:31,271 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:31,276 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:31,277 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service alert-manager's template file
2020-05-10 21:22:31,277 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service alert-manager's configuration.
2020-05-10 21:22:31,277 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service alert-manager.
2020-05-10 21:22:31,277 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service alert-manager is created.
2020-05-10 21:22:31,277 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/alert-manager/deploy/alert-manager.yaml.template.
2020-05-10 21:22:31,277 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/alert-manager/deploy/alert-manager.yaml.
2020-05-10 21:22:31,282 [INFO] - root : It is not a service deploy file! Only support ['DaemonSet', 'Deployment', 'StatefulSets', 'Pod']
2020-05-10 21:22:31,282 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/alert-manager/deploy/alert-configmap.yaml.template.
2020-05-10 21:22:31,282 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/alert-manager/deploy/alert-configmap.yaml.
2020-05-10 21:22:31,286 [INFO] - root : It is not a service deploy file! Only support ['DaemonSet', 'Deployment', 'StatefulSets', 'Pod']
2020-05-10 21:22:31,286 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/alert-manager/deploy/start.sh.template.
2020-05-10 21:22:31,286 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/alert-manager/deploy/start.sh.
2020-05-10 21:22:31,287 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service alert-manager is generated.
2020-05-10 21:22:31,287 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ alert-manager ]
2020-05-10 21:22:31,287 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service alert-manager's start script.
2020-05-10 21:22:31,292 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:22:31,293 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of alert-manager's service.
2020-05-10 21:22:31,293 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of alert-manager's service have been cleaned up.
2020-05-10 21:22:31,293 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start alert-manager
2020-05-10 21:22:31,293 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:31,295 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:31,295 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service end-to-end-test's template file
2020-05-10 21:22:31,295 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service end-to-end-test's configuration.
2020-05-10 21:22:31,295 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service end-to-end-test.
2020-05-10 21:22:31,295 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service end-to-end-test is created.
2020-05-10 21:22:31,295 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/end-to-end-test/deploy/end-to-end-test.yaml.template.
2020-05-10 21:22:31,295 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/end-to-end-test/deploy/end-to-end-test.yaml.
2020-05-10 21:22:31,299 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service end-to-end-test is generated.
2020-05-10 21:22:31,299 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ end-to-end-test ]
2020-05-10 21:22:31,299 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service end-to-end-test's start script.
2020-05-10 21:22:31,304 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:22:31,304 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of end-to-end-test's service.
2020-05-10 21:22:31,304 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of end-to-end-test's service have been cleaned up.
2020-05-10 21:22:31,304 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start end-to-end-test
2020-05-10 21:22:31,304 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:31,310 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:22:31,310 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service prometheus's template file
2020-05-10 21:22:31,310 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service prometheus's configuration.
2020-05-10 21:22:31,310 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service prometheus.
2020-05-10 21:22:31,310 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service prometheus is created.
2020-05-10 21:22:31,310 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/prometheus-configmap.yaml.template.
2020-05-10 21:22:31,310 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/prometheus-configmap.yaml.
2020-05-10 21:22:31,316 [INFO] - root : It is not a service deploy file! Only support ['DaemonSet', 'Deployment', 'StatefulSets', 'Pod']
2020-05-10 21:22:31,316 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/prometheus-deployment.yaml.template.
2020-05-10 21:22:31,316 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/prometheus-deployment.yaml.
2020-05-10 21:22:31,331 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/start.sh.template.
2020-05-10 21:22:31,331 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/start.sh.
2020-05-10 21:22:31,332 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/delete.yaml.template.
2020-05-10 21:22:31,332 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/prometheus/deploy/delete.yaml.
2020-05-10 21:22:31,334 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service prometheus is generated.
2020-05-10 21:22:31,334 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ prometheus ]
2020-05-10 21:22:31,334 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service prometheus's start script.
W0510 21:22:31.380672 19703 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/prometheus-alert created
configmap/prometheus-configmap created
deployment.apps/prometheus-deployment created
prometheus is not ready yet. Please wait for a moment!
prometheus is not ready yet. Please wait for a moment!
prometheus is ready!
2020-05-10 21:23:01,993 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of prometheus's service.
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of prometheus's service have been cleaned up.
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start prometheus
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service grafana's template file
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service grafana's configuration.
2020-05-10 21:23:01,994 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service grafana.
2020-05-10 21:23:01,995 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service grafana is created.
2020-05-10 21:23:01,995 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana.yaml.template.
2020-05-10 21:23:01,995 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana.yaml.
2020-05-10 21:23:02,007 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-clusterview-dashboard.json.template.
2020-05-10 21:23:02,007 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-clusterview-dashboard.json.
2020-05-10 21:23:02,016 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-nodeview-dashboard.json.template.
2020-05-10 21:23:02,017 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-nodeview-dashboard.json.
2020-05-10 21:23:02,025 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-jobview-dashboard.json.template.
2020-05-10 21:23:02,025 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-jobview-dashboard.json.
2020-05-10 21:23:02,032 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-taskroleview-dashboard.json.template.
2020-05-10 21:23:02,032 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-taskroleview-dashboard.json.
2020-05-10 21:23:02,040 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-taskview-dashboard.json.template.
2020-05-10 21:23:02,040 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-taskview-dashboard.json.
2020-05-10 21:23:02,050 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/prom-datasource.json.template.
2020-05-10 21:23:02,050 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/prom-datasource.json.
2020-05-10 21:23:02,051 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-serviceview-dashboard.json.template.
2020-05-10 21:23:02,051 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/grafana/deploy/grafana-configuration/pai-serviceview-dashboard.json.
2020-05-10 21:23:02,057 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service grafana is generated.
2020-05-10 21:23:02,057 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ grafana ]
2020-05-10 21:23:02,057 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service grafana's start script.
W0510 21:23:02.177449 21983 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client.
configmap/grafana-configuration created
deployment.apps/grafana created
grafana is not ready yet. Please wait for a moment!
grafana is ready!
2020-05-10 21:23:22,785 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:23:22,786 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of grafana's service.
2020-05-10 21:23:22,786 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of grafana's service have been cleaned up.
2020-05-10 21:23:22,786 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start grafana
2020-05-10 21:23:22,786 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:22,789 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:22,789 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service node-exporter's template file
2020-05-10 21:23:22,789 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service node-exporter's configuration.
2020-05-10 21:23:22,789 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service node-exporter.
2020-05-10 21:23:22,789 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service node-exporter is created.
2020-05-10 21:23:22,789 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/node-exporter/deploy/node-exporter.yaml.template.
2020-05-10 21:23:22,789 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/node-exporter/deploy/node-exporter.yaml.
2020-05-10 21:23:22,807 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service node-exporter is generated.
2020-05-10 21:23:22,807 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ node-exporter ]
2020-05-10 21:23:22,807 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service node-exporter's start script.
daemonset.apps/node-exporter created
node-exporter is not ready yet. Please wait for a moment!
node-exporter is ready!
2020-05-10 21:23:43,249 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:23:43,250 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of node-exporter's service.
2020-05-10 21:23:43,250 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of node-exporter's service have been cleaned up.
2020-05-10 21:23:43,250 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start node-exporter
2020-05-10 21:23:43,250 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:43,253 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:43,253 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service pylon's template file
2020-05-10 21:23:43,253 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service pylon's configuration.
2020-05-10 21:23:43,253 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service pylon.
2020-05-10 21:23:43,253 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service pylon is created.
2020-05-10 21:23:43,253 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/pylon/deploy/pylon.yaml.template.
2020-05-10 21:23:43,253 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/pylon/deploy/pylon.yaml.
2020-05-10 21:23:43,273 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/pylon/deploy/pylon-service.yaml.template.
2020-05-10 21:23:43,273 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/pylon/deploy/pylon-service.yaml.
2020-05-10 21:23:43,276 [INFO] - root : It is not a service deploy file! Only support ['DaemonSet', 'Deployment', 'StatefulSets', 'Pod']
2020-05-10 21:23:43,276 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/pylon/deploy/start.sh.template.
2020-05-10 21:23:43,276 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/pylon/deploy/start.sh.
2020-05-10 21:23:43,278 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service pylon is generated.
2020-05-10 21:23:43,278 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ pylon ]
2020-05-10 21:23:43,278 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service pylon's start script.
daemonset.apps/pylon-ds created
pylon is ready!
2020-05-10 21:23:53,679 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:23:53,680 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of pylon's service.
2020-05-10 21:23:53,680 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of pylon's service have been cleaned up.
2020-05-10 21:23:53,680 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start pylon
2020-05-10 21:23:53,680 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:53,682 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:23:53,683 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service watchdog's template file
2020-05-10 21:23:53,683 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service watchdog's configuration.
2020-05-10 21:23:53,683 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service watchdog.
2020-05-10 21:23:53,683 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service watchdog is created.
2020-05-10 21:23:53,683 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/watchdog/deploy/watchdog.yaml.template.
2020-05-10 21:23:53,683 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/watchdog/deploy/watchdog.yaml.
2020-05-10 21:23:53,698 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service watchdog is generated.
2020-05-10 21:23:53,698 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ watchdog ]
2020-05-10 21:23:53,698 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service watchdog's start script.
deployment.apps/watchdog created
watchdog is not ready yet. Please wait for a moment!
watchdog is not ready yet. Please wait for a moment!
watchdog is not ready yet. Please wait for a moment!
watchdog is not ready yet. Please wait for a moment!
watchdog is ready!
2020-05-10 21:24:44,175 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:24:44,175 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of watchdog's service.
2020-05-10 21:24:44,175 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of watchdog's service have been cleaned up.
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start watchdog
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service cleaner's template file
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service cleaner's configuration.
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service cleaner.
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service cleaner is created.
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/cleaner/deploy/cleaner.yaml.template.
2020-05-10 21:24:44,176 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/cleaner/deploy/cleaner.yaml.
2020-05-10 21:24:44,192 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service cleaner is generated.
2020-05-10 21:24:44,192 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ cleaner ]
2020-05-10 21:24:44,192 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service cleaner's start script.
2020-05-10 21:24:44,197 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:24:44,198 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of cleaner's service.
2020-05-10 21:24:44,198 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of cleaner's service have been cleaned up.
2020-05-10 21:24:44,198 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start cleaner
2020-05-10 21:24:44,198 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:24:44,200 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
2020-05-10 21:24:44,201 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to generate service job-exporter's template file
2020-05-10 21:24:44,201 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Begin to generate the template file in service job-exporter's configuration.
2020-05-10 21:24:44,201 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Create template mapper for service job-exporter.
2020-05-10 21:24:44,201 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Done. Template mapper for service job-exporter is created.
2020-05-10 21:24:44,201 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Generate the template file /pai/deployment/paiLibrary/paiService/../../../src/job-exporter/deploy/job-exporter.yaml.template.
2020-05-10 21:24:44,201 [INFO] - deployment.paiLibrary.paiService.service_template_generate : Save the generated file to /pai/deployment/paiLibrary/paiService/../../../src/job-exporter/deploy/job-exporter.yaml.
2020-05-10 21:24:44,223 [INFO] - deployment.paiLibrary.paiService.service_template_generate : The template file of service job-exporter is generated.
2020-05-10 21:24:44,223 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to start service: [ job-exporter ]
2020-05-10 21:24:44,223 [INFO] - deployment.paiLibrary.paiService.service_start : Begin to execute service job-exporter's start script.
daemonset.apps/job-exporter created
job-exporter is not ready yet. Please wait for a moment!
job-exporter is not ready yet. Please wait for a moment!
job-exporter is ready!
2020-05-10 21:25:04,842 [INFO] - deployment.paiLibrary.paiService.service_management_start : Begin to clean all service's generated template file
2020-05-10 21:25:04,842 [INFO] - deployment.paiLibrary.paiService.service_template_clean : Begin to delete the generated template of job-exporter's service.
2020-05-10 21:25:04,842 [INFO] - deployment.paiLibrary.paiService.service_template_clean : The generated template files of job-exporter's service have been cleaned up.
2020-05-10 21:25:04,842 [INFO] - deployment.paiLibrary.paiService.service_management_start : Successfully start job-exporter
2020-05-10 21:25:04,842 [INFO] - deployment.paiLibrary.paiService.service_management_start : -----------------------------------------------------------
root@ecs-45ae:/pai#
root@ecs-45ae:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
081f4dff50a2 openpai/job-exporter "python /job_exporte…" About a minute ago Up About a minute k8s_job-exporter_job-exporter-tl4sl_default_9ccc160e-92c1-11ea-928b-fa163eb47bda_0
b2b51796b1f6 openpai/pause-amd64:3.0 "/pause" About a minute ago Up About a minute k8s_POD_job-exporter-tl4sl_default_9ccc160e-92c1-11ea-928b-fa163eb47bda_0
ca0bbc122f00 openpai/watchdog "python /watchdog.py…" 2 minutes ago Up 2 minutes k8s_watchdog_watchdog-99c658b56-slvww_default_7e97c1bf-92c1-11ea-928b-fa163eb47bda_0
575dc60f20c4 openpai/pause-amd64:3.0 "/pause" 2 minutes ago Up 2 minutes k8s_POD_watchdog-99c658b56-slvww_default_7e97c1bf-92c1-11ea-928b-fa163eb47bda_0
47cf638e820c openpai/pylon "/bin/sh -c 'python …" 2 minutes ago Up 2 minutes k8s_pylon_pylon-ds-5mw7s_default_785d86e3-92c1-11ea-928b-fa163eb47bda_0
36ae36870756 openpai/pause-amd64:3.0 "/pause" 2 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:50070->50070/tcp k8s_POD_pylon-ds-5mw7s_default_785d86e3-92c1-11ea-928b-fa163eb47bda_0
85da550437f3 openpai/node-exporter "/bin/node_exporter …" 2 minutes ago Up 2 minutes k8s_node-exporter_node-exporter-lfnst_default_6c2a5a8c-92c1-11ea-928b-fa163eb47bda_0
7ce6cd0ad9fe openpai/pause-amd64:3.0 "/pause" 2 minutes ago Up 2 minutes k8s_POD_node-exporter-lfnst_default_6c2a5a8c-92c1-11ea-928b-fa163eb47bda_0
817be4a91169 openpai/grafana "/usr/local/bin/run-…" 3 minutes ago Up 3 minutes k8s_grafana_grafana-695c89f88-f75nr_default_5ffabd7b-92c1-11ea-928b-fa163eb47bda_0
8b26b355dc4b openpai/pause-amd64:3.0 "/pause" 3 minutes ago Up 3 minutes k8s_POD_grafana-695c89f88-f75nr_default_5ffabd7b-92c1-11ea-928b-fa163eb47bda_0
7aa3561f956c openpai/prometheus "/bin/prometheus --c…" 3 minutes ago Up 3 minutes k8s_prometheus_prometheus-deployment-dc756b66f-7t8wz_default_4d9c5ef2-92c1-11ea-928b-fa163eb47bda_0
80b584a960d5 openpai/pause-amd64:3.0 "/pause" 3 minutes ago Up 3 minutes k8s_POD_prometheus-deployment-dc756b66f-7t8wz_default_4d9c5ef2-92c1-11ea-928b-fa163eb47bda_0
35711ffbd8b7 openpai/webportal "npm start" 3 minutes ago Up 3 minutes k8s_webportal_webportal-ds-j5jxw_default_41445cb0-92c1-11ea-928b-fa163eb47bda_0
0265c4fb4be6 openpai/pause-amd64:3.0 "/pause" 4 minutes ago Up 4 minutes 0.0.0.0:9286->8080/tcp k8s_POD_webportal-ds-j5jxw_default_41445cb0-92c1-11ea-928b-fa163eb47bda_0
532840d8d135 openpai/rest-server "npm start" 4 minutes ago Up 4 minutes k8s_rest-server_rest-server-ds-rddsr_default_351299da-92c1-11ea-928b-fa163eb47bda_0
cd8678905113 openpai/pause-amd64:3.0 "/pause" 4 minutes ago Up 4 minutes 0.0.0.0:9186->8080/tcp k8s_POD_rest-server-ds-rddsr_default_351299da-92c1-11ea-928b-fa163eb47bda_0
f473e2db213d openpai/yarn-frameworklauncher "/usr/local/start.sh" 4 minutes ago Up 4 minutes k8s_yarn-frameworklauncher_yarn-frameworklauncher-ds-7vhbs_default_2ecfb427-92c1-11ea-928b-fa163eb47bda_0
1ee8248ca12e openpai/pause-amd64:3.0 "/pause" 4 minutes ago Up 4 minutes k8s_POD_yarn-frameworklauncher-ds-7vhbs_default_2ecfb427-92c1-11ea-928b-fa163eb47bda_0
7df18abe5601 openpai/hadoop-run "/usr/local/start.sh" 5 minutes ago Up 5 minutes k8s_hadoop-jobhistory-service_hadoop-jobhistory-service-t2xmd_default_166490f2-92c1-11ea-928b-fa163eb47bda_0
3061f5b9e170 openpai/pause-amd64:3.0 "/pause" 5 minutes ago Up 5 minutes k8s_POD_hadoop-jobhistory-service-t2xmd_default_166490f2-92c1-11ea-928b-fa163eb47bda_0
556486ac6d45 openpai/hadoop-run "/usr/local/start.sh" 5 minutes ago Up 5 minutes k8s_hadoop-node-manager_hadoop-node-manager-ds-59tcf_default_0427d6f0-92c1-11ea-928b-fa163eb47bda_0
e765036ec2fa openpai/pause-amd64:3.0 "/pause" 5 minutes ago Up 5 minutes k8s_POD_hadoop-node-manager-ds-59tcf_default_0427d6f0-92c1-11ea-928b-fa163eb47bda_0
176a6e40f191 openpai/yarn-exporter "python3 /usr/local/…" 6 minutes ago Up 5 minutes k8s_yarn-exporter_hadoop-resource-manager-ds-c7hmn_default_ebefe26f-92c0-11ea-928b-fa163eb47bda_0
d7d024f0dcd7 openpai/hadoop-run "/usr/local/start.sh" 6 minutes ago Up 6 minutes k8s_hadoop-resource-manager_hadoop-resource-manager-ds-c7hmn_default_ebefe26f-92c0-11ea-928b-fa163eb47bda_0
0b8de15ca6bd openpai/pause-amd64:3.0 "/pause" 6 minutes ago Up 6 minutes k8s_POD_hadoop-resource-manager-ds-c7hmn_default_ebefe26f-92c0-11ea-928b-fa163eb47bda_0
363225e0c15d openpai/hadoop-run "/usr/local/start.sh" 6 minutes ago Up 6 minutes k8s_hadoop-data-node_hadoop-data-node-ds-498p6_default_dfac4145-92c0-11ea-928b-fa163eb47bda_0
70085a8dff81 openpai/pause-amd64:3.0 "/pause" 6 minutes ago Up 6 minutes k8s_POD_hadoop-data-node-ds-498p6_default_dfac4145-92c0-11ea-928b-fa163eb47bda_0
3691c1e79ac7 openpai/hadoop-run "/usr/local/start.sh" 7 minutes ago Up 7 minutes k8s_hadoop-name-node_hadoop-name-node-ds-sprng_default_cd7164a2-92c0-11ea-928b-fa163eb47bda_0
0ef1a16a7a63 openpai/pause-amd64:3.0 "/pause" 7 minutes ago Up 7 minutes k8s_POD_hadoop-name-node-ds-sprng_default_cd7164a2-92c0-11ea-928b-fa163eb47bda_0
e35b6b642c3f openpai/zookeeper "/bin/bash /zk-confi…" 7 minutes ago Up 7 minutes k8s_zookeeper_zookeeper-ds-z775f_default_c0d1e82c-92c0-11ea-928b-fa163eb47bda_0
7a6be5704a41 openpai/pause-amd64:3.0 "/pause" 7 minutes ago Up 7 minutes k8s_POD_zookeeper-ds-z775f_default_c0d1e82c-92c0-11ea-928b-fa163eb47bda_0
d3ec3b74ee59 openpai/drivers-384.111 "bash -x ./install-a…" 7 minutes ago Up 7 minutes k8s_nvidia-drivers_drivers-one-shot-xw2rr_default_ae9711e4-92c0-11ea-928b-fa163eb47bda_0
b768e661ca80 openpai/pause-amd64:3.0 "/pause" 8 minutes ago Up 8 minutes k8s_POD_drivers-one-shot-xw2rr_default_ae9711e4-92c0-11ea-928b-fa163eb47bda_0
194c83c61394 openpai/kubernetes-dashboard-amd64 "/dashboard --insecu…" 11 minutes ago Up 11 minutes k8s_kubernetes-dashboard_kubernetes-dashboard-7b76b6c9f8-c5sds_kube-system_2c7a43d4-92c0-11ea-928b-fa163eb47bda_0
77fff27dc704 openpai/hyperkube "/hyperkube proxy --…" 11 minutes ago Up 11 minutes k8s_kube-proxy-container_kube-proxy-2knjf_kube-system_2c89e20d-92c0-11ea-928b-fa163eb47bda_0
4ee13d022a85 openpai/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_kube-proxy-2knjf_kube-system_2c89e20d-92c0-11ea-928b-fa163eb47bda_0
a81d9e338d94 openpai/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_kubernetes-dashboard-7b76b6c9f8-c5sds_kube-system_2c7a43d4-92c0-11ea-928b-fa163eb47bda_0
2b5910e2f3fe openpai/etcd "/usr/local/bin/etcd…" 11 minutes ago Up 11 minutes k8s_etcd-container_etcd-server-192.168.114.178_default_bd20e60111fc0e4eb5e13aeadfb876b0_0
8a57d18e7cb7 openpai/kube-scheduler "/usr/local/bin/kube…" 11 minutes ago Up 11 minutes k8s_kube-scheduler_kube-scheduler-192.168.114.178_kube-system_b5501d2f80f05f1c9f979d97f3ba6be3_0
81ec79a672c1 openpai/kube-apiserver "/usr/local/bin/kube…" 11 minutes ago Up 11 minutes k8s_apiserver-container_kube-apiserver-192.168.114.178_kube-system_a75ed8c4c40dc62841875e4fac63194a_0
fcbe376f8a4a openpai/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_etcd-server-192.168.114.178_default_bd20e60111fc0e4eb5e13aeadfb876b0_0
6f2c5fda5caa openpai/kube-controller-manager "/usr/local/bin/kube…" 11 minutes ago Up 11 minutes k8s_kube-controller-manager_kube-controller-manager-192.168.114.178_kube-system_66563e753ab05eaf2fe5e86faf68efe8_0
c3dd033740f5 openpai/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_kube-apiserver-192.168.114.178_kube-system_a75ed8c4c40dc62841875e4fac63194a_0
e981abd503e6 openpai/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_kube-scheduler-192.168.114.178_kube-system_b5501d2f80f05f1c9f979d97f3ba6be3_0
9dfdde9a1f07 openpai/pause-amd64:3.0 "/pause" 11 minutes ago Up 11 minutes k8s_POD_kube-controller-manager-192.168.114.178_kube-system_66563e753ab05eaf2fe5e86faf68efe8_0
81b01a662f4a openpai/hyperkube:v1.9.9 "/hyperkube kubelet …" 11 minutes ago Up 11 minutes kubelet
root@ecs-45ae:~#
root@ecs-45ae:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest bf756fb1ae65 4 months ago 13.3kB
openpai/job-exporter latest 460254b6072a 14 months ago 1.06GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/job-exporter latest 460254b6072a 14 months ago 1.06GB
openpai/watchdog latest 0f82b772a9d9 14 months ago 992MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/watchdog latest 0f82b772a9d9 14 months ago 992MB
openpai/rest-server latest 47f6bf04be6c 14 months ago 1.04GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/rest-server latest 47f6bf04be6c 14 months ago 1.04GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/webportal latest 3c501b3a40d1 14 months ago 1.51GB
openpai/webportal latest 3c501b3a40d1 14 months ago 1.51GB
openpai/yarn-frameworklauncher latest 591fc12748ce 15 months ago 1.93GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/yarn-frameworklauncher latest 591fc12748ce 15 months ago 1.93GB
openpai/end-to-end-test latest 4fd69f739b5e 15 months ago 2.1GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/end-to-end-test latest 4fd69f739b5e 15 months ago 2.1GB
openpai/hadoop-run latest d596e208c837 15 months ago 1.74GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/hadoop-run latest d596e208c837 15 months ago 1.74GB
openpai/cleaning-image latest 806775ef0540 15 months ago 967MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/cleaning-image latest 806775ef0540 15 months ago 967MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/zookeeper latest 4de8da83bded 15 months ago 917MB
openpai/zookeeper latest 4de8da83bded 15 months ago 917MB
openpai/base-image latest 7fe7fc9abc00 15 months ago 874MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/base-image latest 7fe7fc9abc00 15 months ago 874MB
openpai/cleaner latest 2fc855208a24 15 months ago 1.03GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/cleaner latest 2fc855208a24 15 months ago 1.03GB
openpai/yarn-exporter latest 8d0e652c4e8a 15 months ago 942MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/yarn-exporter latest 8d0e652c4e8a 15 months ago 942MB
openpai/grafana latest 0e8a285d92b7 15 months ago 283MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/grafana latest 0e8a285d92b7 15 months ago 283MB
openpai/drivers-384.111 latest 02908196dcc3 15 months ago 3.07GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/drivers-384.111 latest 02908196dcc3 15 months ago 3.07GB
openpai/drivers-390.25 latest 3ad4a1f2f46b 15 months ago 3.76GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/drivers-390.25 latest 3ad4a1f2f46b 15 months ago 3.76GB
openpai/drivers-410.73 latest 836163c091ef 15 months ago 3.88GB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/drivers-410.73 latest 836163c091ef 15 months ago 3.88GB
openpai/pylon latest 4510b0b6c8f2 15 months ago 520MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/pylon latest 4510b0b6c8f2 15 months ago 520MB
openpai/hyperkube v1.9.9 06696cbe4586 22 months ago 652MB
openpai/kube-controller-manager v1.9.9 2f4813f0ccc3 22 months ago 140MB
openpai/kube-apiserver v1.9.9 09a56741e8c8 22 months ago 213MB
openpai/kube-scheduler v1.9.9 74cea6a9ab4d 22 months ago 63.7MB
openpai/node-exporter latest 188af75e2de0 24 months ago 22.9MB
swr.cn-north-1.myhuaweicloud.com/xsjzdrxsjzdr/node-exporter latest 188af75e2de0 24 months ago 22.9MB
openpai/etcd 3.2.17 0d36cbbcc544 2 years ago 229MB
openpai/kubernetes-dashboard-amd64 v1.8.3 0c60bcf89900 2 years ago 102MB
openpai/prometheus latest c8ecf7c719c1 2 years ago 112MB
openpai/pause-amd64 3.0 99e59f495ffa 4 years ago 747kB
root@ecs-45ae:~#
root@ecs-45ae:~# kubectl get nodes
NAME AGE
192.168.114.178 15m
root@ecs-45ae:~# kubectl get all --all-namespaces
NAMESPACE NAME AGE
default pod/batch-job-hadoop-vkcb2 6m44s
default pod/drivers-one-shot-xw2rr 10m
default pod/etcd-server-192.168.114.178 14m
default pod/grafana-695c89f88-f75nr 5m21s
default pod/hadoop-data-node-ds-498p6 8m56s
default pod/hadoop-jobhistory-service-t2xmd 7m24s
default pod/hadoop-name-node-ds-sprng 9m27s
default pod/hadoop-node-manager-ds-59tcf 7m55s
default pod/hadoop-resource-manager-ds-c7hmn 8m36s
default pod/job-exporter-tl4sl 3m39s
default pod/node-exporter-lfnst 5m1s
default pod/prometheus-deployment-dc756b66f-7t8wz 5m52s
default pod/pylon-ds-5mw7s 4m40s
default pod/rest-server-ds-rddsr 6m33s
default pod/watchdog-99c658b56-slvww 4m30s
default pod/webportal-ds-j5jxw 6m13s
default pod/yarn-frameworklauncher-ds-7vhbs 6m44s
default pod/zookeeper-ds-z775f 9m48s
kube-system pod/kube-apiserver-192.168.114.178 15m
kube-system pod/kube-controller-manager-192.168.114.178 15m
kube-system pod/kube-proxy-2knjf 13m
kube-system pod/kube-scheduler-192.168.114.178 14m
kube-system pod/kubernetes-dashboard-7b76b6c9f8-c5sds 13m
NAMESPACE NAME AGE
default service/kubernetes 15m
kube-system service/kubernetes-dashboard 13m
NAMESPACE NAME AGE
default daemonset.extensions/drivers-one-shot 10m
default daemonset.extensions/hadoop-data-node-ds 8m56s
default daemonset.extensions/hadoop-jobhistory-service 7m24s
default daemonset.extensions/hadoop-name-node-ds 9m27s
default daemonset.extensions/hadoop-node-manager-ds 7m55s
default daemonset.extensions/hadoop-resource-manager-ds 8m36s
default daemonset.extensions/job-exporter 3m39s
default daemonset.extensions/node-exporter 5m1s
default daemonset.extensions/pylon-ds 4m40s
default daemonset.extensions/rest-server-ds 6m33s
default daemonset.extensions/webportal-ds 6m13s
default daemonset.extensions/yarn-frameworklauncher-ds 6m44s
default daemonset.extensions/zookeeper-ds 9m48s
kube-system daemonset.extensions/kube-proxy 13m
NAMESPACE NAME AGE
default deployment.extensions/grafana 5m21s
default deployment.extensions/prometheus-deployment 5m52s
default deployment.extensions/watchdog 4m30s
kube-system deployment.extensions/kubernetes-dashboard 13m
NAMESPACE NAME AGE
default replicaset.extensions/grafana-695c89f88 5m21s
default replicaset.extensions/prometheus-deployment-dc756b66f 5m52s
default replicaset.extensions/watchdog-99c658b56 4m30s
kube-system replicaset.extensions/kubernetes-dashboard-7b76b6c9f8 13m
NAMESPACE NAME AGE
default daemonset.apps/drivers-one-shot 10m
default daemonset.apps/hadoop-data-node-ds 8m56s
default daemonset.apps/hadoop-jobhistory-service 7m24s
default daemonset.apps/hadoop-name-node-ds 9m27s
default daemonset.apps/hadoop-node-manager-ds 7m55s
default daemonset.apps/hadoop-resource-manager-ds 8m36s
default daemonset.apps/job-exporter 3m39s
default daemonset.apps/node-exporter 5m1s
default daemonset.apps/pylon-ds 4m40s
default daemonset.apps/rest-server-ds 6m33s
default daemonset.apps/webportal-ds 6m13s
default daemonset.apps/yarn-frameworklauncher-ds 6m44s
default daemonset.apps/zookeeper-ds 9m48s
kube-system daemonset.apps/kube-proxy 13m
NAMESPACE NAME AGE
default deployment.apps/grafana 5m21s
default deployment.apps/prometheus-deployment 5m52s
default deployment.apps/watchdog 4m30s
kube-system deployment.apps/kubernetes-dashboard 13m
NAMESPACE NAME AGE
default replicaset.apps/grafana-695c89f88 5m21s
default replicaset.apps/prometheus-deployment-dc756b66f 5m52s
default replicaset.apps/watchdog-99c658b56 4m30s
kube-system replicaset.apps/kubernetes-dashboard-7b76b6c9f8 13m
NAMESPACE NAME AGE
default job.batch/batch-job-hadoop 6m44s
root@ecs-45ae:~# kubectl get nodes
NAME AGE
192.168.114.178 15m
root@ecs-45ae:~# kubectl get all namespaces
error: you must specify only one resource
root@ecs-45ae:~# kubectl get namespace
NAME AGE
default 15m
kube-public 15m
kube-system 15m
pai-user 6m45s
root@ecs-45ae:~# kubectl get pods -n pai-user
No resources found in pai-user namespace.
root@ecs-45ae:~# kubectl get pods -n kube-system
NAME AGE
kube-apiserver-192.168.114.178 15m
kube-controller-manager-192.168.114.178 15m
kube-proxy-2knjf 14m
kube-scheduler-192.168.114.178 15m
kubernetes-dashboard-7b76b6c9f8-c5sds 14m
root@ecs-45ae:~# kubectl get pods -n kube-public
No resources found in kube-public namespace.
root@ecs-45ae:~# kubectl get pods
NAME AGE
batch-job-hadoop-vkcb2 7m44s
drivers-one-shot-xw2rr 11m
etcd-server-192.168.114.178 15m
grafana-695c89f88-f75nr 6m21s
hadoop-data-node-ds-498p6 9m56s
hadoop-jobhistory-service-t2xmd 8m24s
hadoop-name-node-ds-sprng 10m
hadoop-node-manager-ds-59tcf 8m55s
hadoop-resource-manager-ds-c7hmn 9m36s
job-exporter-tl4sl 4m39s
node-exporter-lfnst 6m1s
prometheus-deployment-dc756b66f-7t8wz 6m52s
pylon-ds-5mw7s 5m40s
rest-server-ds-rddsr 7m33s
watchdog-99c658b56-slvww 5m30s
webportal-ds-j5jxw 7m13s
yarn-frameworklauncher-ds-7vhbs 7m44s
zookeeper-ds-z775f 10m
root@ecs-45ae:~#
微软OpenPai平台部署安装(kubernetes 大数据和深度学习平台安装部署)相关推荐
- 基于大数据与深度学习的自然语言对话
基于大数据与深度学习的自然语言对话 发表于2015-12-04 09:44| 7989次阅读| 来源<程序员>电子刊| 5 条评论| 作者李航.吕正东.尚利峰 大数据深度学习自然语言处理自 ...
- 人工智能，大数据，深度学习，机器学习（百度云581G学习资料免费分享）
人工智能视频学习 链接:https://pan.baidu.com/s/14HwJZhcYj6L7l3ROUMmqsQ 提取码:264a 复制这段内容后打开百度网盘手机App,操作更方便哦 机器学习 ...
- 谷俊丽：基于大数据的深度学习
嘉宾介绍:谷俊丽,博士学历,毕业于清华大学-美国University of Illinois Urbana-champaign大学,在清华期间作为核心人员研发过超长指令字数字信号处理器,在美期间曾参与 ...
- 【转】谷俊丽：基于大数据的深度学习
对于我这种小白,还比较通俗易懂,顺便也介绍了深度学习一些概况,留着有用! 嘉宾介绍:谷俊丽,博士学历,毕业于清华大学-美国University of Illinois Urbana-champaign ...
- 大数据和深度学习之间，主要是什么关系？
简单来说: 1)深度学习(DeepLearning)只是机器学习(MachineLearning)的一种类别,一个子领域.机器学习>深度学习 2)大数据(BigData)不是具体的方法,甚至不算 ...
- 【网络流量识别】【深度学习】【四】DNN、GBT和RF—利用大数据和深度学习技术进行入侵检测
本文发表于2019年4月,ACM东南会议纪要,作者为奥萨马·费克等人,现收录于ACM网站. 原文题目:使用大数据和深度学习技术进行入侵检测 原文链接:使用大数据和深度学习技术进行入侵检测|2019年A ...
- 从大数据到深度学习，这些年度开源“新秀”你可用过？
六月 北京 | 高性能计算之GPU CUDA培训 6月22-24日三天密集式学习 快速带你入门阅读全文> 转载自:"开源中国"公众号 正文共2308个字,6图,预计阅读时间 ...
- 1. 大数据 机器学习 深度学习 推荐系统 学习路线
文章目录 思维导图下载 大数据基础 linux NGINX负载均衡 Zookeeper hadoop生态 hadoop hive hbase 大数据数据仓库 数据仓库基础 日志收集系统(Flume) ...
- 大数据毕设 - 深度学习车牌识别系统（python 机器视觉 opencv）
文章目录 1 前言 1 课题背景 2 效果演示 2.1 图片检测识别 2.2视频检测识别 3 车牌检测与识别 4 HyperLPR库 4.1 简介 4.2 特点 4.3 HyperLPR的检测流程 4 ...
最新文章
- sklearn中的xgboost_xgboost来了
- Eigen矩阵运算的混淆问题
- C++ leetcode 26. 删除排序数组中的重复项 给定一个排序数组，你需要在 原地 删除重复出现的元素，使得每个元素只出现一次，返回移除后数组的新长度。
- python进行数据分析，学习笔记 第8章（1）
- 卡尔曼滤波---实例讲解
- POJ 3660 Cow Contest（传递闭包floyed算法）
- LeetCode 1861. 旋转盒子（前缀和）
- Spring Boot————ApplicationListener实现逃课事件监听
- Ubuntu 12.04 Server OpenStack Havana多节点(OVS+GRE)安装
- 使用LXCFS实现容器资源的视图隔离
- R语言学习笔记（六）回归分析
- OpenCV精进之路（一）：图像处理知识点串烧
- LuoguP1041 传染病控制
- 打开netlogo model 出现failed to launch JVM
- Perl Unicode全攻略
- 已解决：excel求平均值时出现div/0 如何处理
- C语言指针中 *p 和 p 的区别
- [BZOJ2844]albus就是要第一个出场
- Delete数据如何恢复
- 龙格库塔法的基本原理