TF实战丨使用Vagrant安装Tungsten Fabric
本文为苏宁网络架构师陈刚的原创文章。
01 准备测试机
在16G的笔记本没跑起来,就干脆拼凑了一台游戏工作室级别的机器:双路E5-2860v3 CPU,24核48线程,128G DDR4 ECC内存,NVME盘 512G。在上面开5个VM,假装是物理服务器。
· 192.16.35.110 deployer
· 192.16.35.111 tf控制器
· 192.16.35.112 openstack服务器,同时也是计算节点
· 192.16.35.113 k8s master
· 192.16.35.114 k8s的Node k01,同时也是ops的计算节点
直接使用vagrant拉镜像会很慢,就先下载下来:
https://cloud.centos.org/centos/7/vagrant/x86_64/images/
下载对应的VirtualBox.box文件。
然后使用命令, 命名为vagrant的box:
vagrant box add centos/7 CentOS-7-x86_64-Vagrant-2004_01.VirtualBox.box
cat << EEOOFF > vagrantfile
###start
#-*- mode: ruby -*-
#vi: set ft=ruby :
Vagrant.require_version ">=2.0.3"#All Vagrant configuration is done below. The "2" in Vagrant.configure
#configures the configuration version (we support older styles for
#backwards compatibility). Please don't change it unless you know what
#you're doing.ENV["LC_ALL"] = "en_US.UTF-8"VAGRANTFILE_API_VERSION = "2"Vagrant.configure("2") do |config|# The most common configuration options are documented and commented below.# For a complete reference, please see the online documentation at# https://docs.vagrantup.com.# Every Vagrant development environment requires a box. You can search for# boxes at https://atlas.hashicorp.com/search.config.vm.box = "geerlingguy/centos7"# config.vbguest.auto_update = false# config.vbguest.no_remote = true config.vm.define "deployer" do | dp |dp.vm.provider "virtualbox" do | v |v.memory = "8000"v.cpus = 2enddp.vm.network "private_network", ip: "192.16.35.110", auto_config: truedp.vm.hostname = "deployer"endconfig.vm.define "tf" do | tf |tf.vm.provider "virtualbox" do | v |v.memory = "64000"v.cpus = 16endtf.vm.network "private_network", ip: "192.16.35.111", auto_config: truetf.vm.hostname = "tf"endconfig.vm.define "ops" do | os |os.vm.provider "virtualbox" do | v |v.memory = "16000"v.cpus = 4endos.vm.network "private_network",ip: "192.16.35.112", auto_config: trueos.vm.hostname = "ops"endconfig.vm.define "k8s" do | k8 |k8.vm.provider "virtualbox" do | v |v.memory = "8000"v.cpus = 2endk8.vm.network "private_network", ip: "192.16.35.113", auto_config: truek8.vm.hostname = "k8s"endconfig.vm.define "k01" do | k1 |k1.vm.provider "virtualbox" do | v |v.memory = "4000"v.cpus = 2endk1.vm.network "private_network", ip: "192.16.35.114", auto_config: truek1.vm.hostname = "k01"endconfig.vm.provision "shell", privileged: true, path: "./setup.sh"endEEOOFFcat << EEOOFF > setup.sh
#!/bin/bash
#
#Setup vagrant vms.
#set -eu#Copy hosts info
cat <<EOF > /etc/hosts
127.0.0.1 localhost
127.0.1.1 vagrant.vm vagrant192.16.35.110 deployer
192.16.35.111 tf
192.16.35.112 ops
192.16.35.113 k8s
192.16.35.114 k01#The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
EOFsystemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat
iptables -P FORWARD ACCEPTswapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab
#swapoff -a && sysctl -w vm.swappiness=0#setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config #modprobe ip_vs_rr
modprobe br_netfilteryum -y update#sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory
#sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
#yum install -y bridge-utils.x86_64
#modprobe bridge
#modprobe br_netfilter
#Setup system varsyum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools vim chrony python python-setuptools python-pip iproute lrzsz tree gityum install -y libguestfs-tools libvirt-python virt-install libvirt ansiblepip install wheel --upgrade -i https://mirrors.aliyun.com/pypi/simple/
pip install pip --upgrade -i https://mirrors.aliyun.com/pypi/simple/
pip install ansible netaddr --upgrade -i https://mirrors.aliyun.com/pypi/simple/#python-urllib3 should be installed before "pip install requests"
#if install failed, pip uninstall urllib3, then reinstall python-urllib3
#pip uninstall -y urllib3 | true
#yum install -y python-urllib3
pip install requests -i https://mirrors.aliyun.com/pypi/simple/systemctl disable libvirtd.service
systemctl disable dnsmasq
systemctl stop libvirtd.service
systemctl stop dnsmasqif [ -d "/root/.ssh" ]; thenrm -rf /root/.ssh
fissh-keygen -q -t rsa -N "" -f ~/.ssh/id_rsacat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod go-rwx ~/.ssh/authorized_keys#
#timedatectl set-timezone Asia/Shanghaiif [ -f "/etc/chrony.conf" ]; thenmv /etc/chrony.conf /etc/chrony.conf.bak
ficat <<EOF > /etc/chrony.confallow 192.16.35.0/24server ntp1.aliyun.com iburstlocal stratum 10logdir /var/log/chronyrtcsyncmakestep 1.0 3driftfile /var/lib/chrony/drift
EOFsystemctl restart chronyd.service
systemctl enable chronyd.serviceecho "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536" >> /etc/security/limits.conf
echo "* hard nproc 65536" >> /etc/security/limits.conf
echo "* soft memlock unlimited" >> /etc/security/limits.conf
echo "* hard memlock unlimited" >> /etc/security/limits.confif [ ! -d "/var/log/journal" ]; thenmkdir /var/log/journal
fiif [ ! -d "/etc/systemd/journald.conf.d" ]; thenmkdir /etc/systemd/journald.conf.d
ficat <<EOF > /etc/systemd/journald.conf.d/99-prophet.conf
[Journal]
Storage=persistentCompress=yesSyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000SystemMaxUse=10GSystemMaxFileSize=200MForwardToSyslog=noEOFsystemctl restart systemd-journaldEEOOFF
02 在所有的节点上安装docker
CentOS
例如:如果pip安装软件的速度很慢,可以考虑使用基于aliyun的pip加速
· 各个节点设置pip加速
mkdir .pip && tee ~/.pip/pip.conf <<-'EOF'[global]trusted-host = mirrors.aliyun.comindex-url = https://mirrors.aliyun.com/pypi/simpleEOF
注意requests包不能在urllib3之后安装,否则会出错:
pip uninstall urllib3
pip uninstall chardet
pip install requests
(这些命令应该都已经在 setup.sh 中执行过了)
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools iproute lrzsz tree git
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum install -y docker-ce
yum -y install epel-release
systemctl daemon-reload
systemctl enable docker
systemctl restart docker
yum install -y chrony
systemctl start chronyd
systemctl enable chronyd
03 拉取并启动Contrail-Kolla-Ansible-Deployer容器
容器的Nightly builds 可以从这里访问: Docker Hub
https://hub.docker.com/r/opencontrailnightly/contrail-kolla-ansible-deployer/tags
例如:
vim /etc/docker/daemon.json
{ "registry-mirrors" : [ "https://hub-mirror.c.163.com","https://registry.docker-cn.com" ] }
systemctl restart dockerexport CAD_IMAGE=opencontrailnightly/contrail-kolla-ansible-deployer:master-latest
docker run -td --net host --name contrail_kolla_ansible_deployer $CAD_IMAGE
04 将配置文件复制到容器
instance.yaml:用于配置Tungsten Fabric集群的模板文件。
https://github.com/Juniper/contrail-ansible-deployer/wiki/Contrail-with-Openstack-Kolla#13-configure-necessary-parameters-configinstancesyaml-under-appropriate-parameters
要获得有关如何配置该文件中所有可用参数的信息,可阅读这里:
https://github.com/Juniper/contrail-ansible-deployer/blob/master/README.md#configuration
cat << EOF > instances.yaml
provider_config: bms: ssh_pwd: vagrant ssh_user: root ntpserver: ntp1.aliyun.com domainsuffix: localinstances: tf: provider: bms ip: 192.16.35.111 roles: config_database: config: control: analytics_database: analytics: webui: ops: provider: bms ip: 192.16.35.112 roles:openstack:openstack_compute: vrouter:PHYSICAL_INTERFACE: enp0s8k8s: provider: bms ip: 192.16.35.113 roles:k8s_master:k8s_node:kubemanager:vrouter:PHYSICAL_INTERFACE: enp0s8k01: provider: bms ip: 192.16.35.114 roles:openstack_compute:k8s_node:vrouter:PHYSICAL_INTERFACE: enp0s8
contrail_configuration: AUTH_MODE: keystone KEYSTONE_AUTH_URL_VERSION: /v3KEYSTONE_AUTH_ADMIN_PASSWORD: vagrantCLOUD_ORCHESTRATOR: openstackCONTRAIL_VERSION: latestUPGRADE_KERNEL: trueENCAP_PRIORITY: "VXLAN,MPLSoUDP,MPLSoGRE"PHYSICAL_INTERFACE: enp0s8
global_configuration:CONTAINER_REGISTRY: opencontrailnightly
kolla_config:kolla_globals: enable_haproxy: no enable_ironic: "no" enable_swift: "no"network_interface: "enp0s8"kolla_passwords: keystone_admin_password: vagrantEOFexport INSTANCES_FILE=instances.yaml
docker cp $INSTANCES_FILE contrail_kolla_ansible_deployer:/root/contrail-ansible-deployer/config/instances.yaml
05 准备好所有节点的环境
除了deployer,我在所有节点上都做了一遍。
正常的做法是建个自己的repository放各种image,实验环境节点少,直接国内下载也很快的。
注意python和python-py这两个包是冲突的,只能安装其中之一,最好先全卸载,再安装其中一个:
pip uninstall docker-py dockerpip install pythonyum -y install python-devel python-subprocess32 python-setuptools python-pippip install --upgrade pipfind / -name *subpro*.egg-infofind / -name *subpro*.egg-info |xargs rm -rfpip install -I sixpip install -I docker-compose
将k8s repository改成阿里的,缺省的Google源太慢或不通:vi
playbooks/roles/k8s/tasks/RedHat.ymlyum_repository:
name: Kubernetes
description: k8s repo
baseurl: https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
gpgkey: https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
repo_gpgcheck: yes
gpgcheck: yes
when: k8s_package_version is defined
playbook中安装这些需要访问海外网站,可以从国内下载,然后改个tag:
k8s.gcr.io/kube-apiserver:v1.14.8
k8s.gcr.io/kube-controller-manager:v1.14.8
k8s.gcr.io/kube-scheduler:v1.14.8
k8s.gcr.io/kube-proxy:v1.14.8
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1
换个方法变通处理一下
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
docker pull coredns/coredns:1.3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3
再重新给下载的打个tag
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8 k8s.gcr.io/kube-apiserver:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8 k8s.gcr.io/kube-controller-manager:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8 k8s.gcr.io/kube-scheduler:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8 k8s.gcr.io/kube-proxy:v1.14.8
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3 k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
06 启动deployer容器,进入其中进行部署
docker start contrail_kolla_ansible_deployer
进入deployer容器:
docker exec -it contrail_kolla_ansible_deployer bashcd /root/contrail-ansible-deployer
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/provision_instances.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/configure_instances.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_openstack.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_k8s.yml
ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_contrail.ymlkubectl taint nodes k8s node-role.kubernetes.io/master-
最后一次kubelet升级到最新,遇到CSI的bug,修改一下配置文件后重启kubelet即可:
After experiencing the same issue, editing /var/lib/kubelet/config.yaml to add:
featureGates: CSIMigration: false
07 安装完成后,建2个VM和容器测试一下
yum install -y gcc python-devel
pip install python-openstackclient
pip install python-ironicclientsource /etc/kolla/kolla-toolbox/admin-openrc.sh
如果openstack命令有如下“queue”的报错,是需要python3:
File "/usr/lib/python2.7/site-packages/openstack/utils.py", line 13, in <module>import queue
ImportError: No module named queue
rm -f /usr/bin/python
ln -s /usr/bin/python3 /usr/bin/python
pip install python-openstackclient
pip install python-ironicclient
yum install -y python3-pipyum install -y gcc python-devel wgetpip install --upgrade setuptoolspip install --ignore-installed python-openstackclient我每次都需要python3,所以干脆也安装了这个:
pip3 install python-openstackclient -i https://mirrors.aliyun.com/pypi/simple/
pip3 install python-ironicclient -i https://mirrors.aliyun.com/pypi/simple/
进入Tungsten Fabric,用浏览器:https://192.16.35.111:8143
进入openstack,用浏览器:https://192.16.35.112
在k8s master上(192.16.35.113):
scp root@192.16.35.114:/opt/cni/bin/contrail-k8s-cni /opt/cni/bin/
mkdir /etc/cni/net.d
scp root@192.16.35.114:/etc/cni/net.d/10-contrail.conf /etc/cni/net.d/10-contrail.conf
wget
https://github.com/cirros-dev/cirros/releases/download/0.4.0/cirros-0.4.0-x86_64-disk.img
官方下载地址
https://download.cirros-cloud.net/
curl -O
https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
wget
http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
wget
http://download.cirros-cloud.net/daily/20161201/cirros-d161201-x86_64-disk.img
(都没有找到带tcpdump的版本)
reboot
source /etc/kolla/kolla-toolbox/admin-openrc.sh
openstack image create cirros --disk-format qcow2 --public --container-format bare --file cirros-0.4.0-x86_64-disk.imgnova flavor-create m1.tiny auto 512 1 1openstack network create net1openstack subnet create --subnet-range 10.1.1.0/24 --network net1 mysubnet1
NET_ID=`openstack network list | grep net1 | awk -F '|' '{print $2}' | tr -d ' '`nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM1
nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM2
进入k8s_master, 192.16.35.113:
yum install -y git
git clone https://github.com/virtualhops/k8s-demo
kubectl create -f k8s-demo/po-ubuntuapp.yml
kubectl create -f k8s-demo/rc-frontend.yml
kubectl expose rc/frontend
kubectl exec -it ubuntuapp curl frontend # many times
参考方案:
https://github.com/Juniper/contrail-ansible-deployer/wiki/%5B-Container-Workflow%5D-Deploying-Contrail-with-OpenStack
Tungsten Fabric实战:对接vMX虚拟路由平台填坑
Tungsten Fabric实战:基于K8s的部署踩坑
TF实战Q&A丨你不理解透,出了问题都不知道怎么弄
TF 实战Q&A丨只在此网中,云深不知处
TF实战丨使用Vagrant安装Tungsten Fabric相关推荐
- Tungsten Fabric知识库丨vRouter内部运行探秘
原文链接: https://github.com/tnaganawa/tungstenfabric-docs/blob/master/TungstenFabricKnowledgeBase.md 作者 ...
- 在K8s上轻松部署Tungsten Fabric的两种方式
第一种:在AWS的K8s上部署TF 首先介绍下如何在AWS上使用Kubernetes编排的Tungsten Fabric集群部署沙盒,15分钟就可以搞定.Tungsten Fabric集群由部署节点. ...
- 如何在OpenStack Kolla上部署Tungsten Fabric(附14个常见的配置问题)
首先,使用contil-kolla-ansible-deployer容器在OpenStack Kolla上部署Tungsten Fabric(注:原文为Contrail,本文以功能一致的Tungste ...
- Tungsten Fabric知识库丨关于OpenStack、K8s、CentOS安装问题的补充
作者:Tatsuya Naganawa 译者:TF编译组 多kube-master部署 3个Tungsten Fabric控制器节点:m3.xlarge(4 vcpu)-> c3.4xlarge ...
- TF Live 直播回放丨Frank Wu:当OpenStack遇到Tungsten Fabric
10岁的OpenStack,已经是开源IaaS世界里的"成年人",自从遇到开源SDN小伙伴Tungsten Fabric,两人便成为闯荡混合多云世界的好搭档. 5月26日,在TF中 ...
- TF实战QA丨这个问题,我以前也遇到过
在TF中文社区,爱折腾的"实战派"们经常探讨有关SDN和Tungsten Fabric的各种问题,我们将其中的精华部分整理出来,形成 " TF Q&A " ...
- Tungsten Fabric知识库丨测试2000个vRouter节点部署
作者:Tatsuya Naganawa 译者:TF编译组 由于GCP允许启动多达5k个节点:),因此vRouter的规模测试程序主要针对该平台来介绍. 话虽如此,AWS也可以使用相同的程序 第一个目标 ...
- Tungsten Fabric入门宝典丨8个典型故障及排查Tips
Tungsten Fabric入门宝典系列文章,来自技术大牛倾囊相授的实践经验,由TF中文社区为您编译呈现,旨在帮助新手深入理解TF的运行.安装.集成.调试等全流程.如果您有相关经验或疑问,欢迎与我们 ...
- Tungsten Fabric SDN — 与 Kubernetes 的集成部署(CN)
目录 文章目录 目录 部署架构 软件版本 部署 Kubernetes & TF 基础环境设置 创建 Deployment instance 执行 Playbooks 环境检查 配置记录 Con ...
最新文章
- SCCM 2012系列1 服务器准备上
- SpringMVC异常报406 (Not Acceptable)的解决办法
- windows10 下 用图片手把手教你 卸载 cygwin
- .NET Core系列 : 2 、project.json 这葫芦里卖的什么药
- x64 stack walking、调用约定、函数参数识别
- WDK中出现的特殊代码
- get方法请求返回一个文件_一键转换多种文件格式,完全免费,总有一个方法适合你...
- python 抽奖器_兄弟连学python (02) ----简易抽奖器
- 蓝桥杯 ADV-105 算法提高 不同单词个数统计
- Mysql中循环拼接参数_利用循环向数据库中插入数据,参数重复的问题
- 排版设计软件QuarkXPress 2022 for mac
- linux w3m命令
- 微信公众号通过a标签打开小程序
- 论文解读:PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
- 【加拿大留学】蒙特利尔中国公派学者 学生学习生活指南【蒙特利尔留学必看,第一次出国必看】
- 移动端和pc端的区别html,pc端是什么意思(PC端和移动端有哪些区别?)
- PPSIG携手100+位contributor,共建飞桨开源社区
- 布尔-施罗德逻辑代数中的命题逻辑-- 布尔逻辑之七(尾篇**)
- xv6-lab2-syscall
- 语法分析器(c++)