开发用的devstack (by quqi99)

作者：张华发表于：2014-05-15
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (http://blog.csdn.net/quqi99 )

导入镜像

镜像是在kvm上做的，做完转成vmware镜像的方法如下：

qemu-img convert -f qcow2 -O vmdk o7k.qcow2 o7k.vmdk

可在win10上下载vmware player (player是免费版), 如 VMware-player-16.1.2-17966106.exe - https://www.vmware.com/cn/products/workstation-player.html
vmware player免费版没有直接导入镜像的功能，但可以这样导入：

创建一个新虚机，"Create a New Virtual Machine -> I will install the operating system later"进去之后随便创建一块硬盘（注意此处选择Store virtual dsik as a single file）
虚机创建好之后，再在虚机的设置中将上面创建的硬盘删除，然后再新建一块硬盘(Hard Disk)，此时可以选择导入已有的镜像(Use an existing virtual disk)
虚机设置使用8G内存及2个vCPU, 其他用默认即可。
另外非常重要的一点是，虚机应该使用UEFI模式而不是BIOS模式。但免费版的vmware player也没有提供此项配置。但我们可以在它的 .vmx 文件（.vmx是虚机文件，.vmdk是虚机的硬盘)的最后一行添加下行即可(可参见：https://www.serverwatch.com/guides/enabling-uefi-on-virtual-machines/）:

firmware="efi"

然后启动虚机后可能没有IP, 那是因为网卡名字可能改化了。需修改netplan的配置文件/etc/netplan/00-installer-config.yaml改成正确的网卡名称(网卡名称可通过ip addr show获取), 然后运行sudo netplan apply即可获得IP

快速运行

sudo mkdir -p /bak/openstack && sudo chown -R $(whoami) /bak/openstack && cd /bak/openstack
sudo chown -R $(whoami) ~/.config
git clone https://github.com/openstack/devstack.git
cd devstack && wget https://raw.githubusercontent.com/zhhuabj/exercise/master/o7k/local.conf
#sudo systemctl stop "devstack@*"
#./clean.sh
./stack.sh
source accrc/admin/admin

[可选] 加速apt与pypi

如果网速很慢，可由一人搭建apt-cacher-ng server, 其他人用它来避免使用apt-get时的网络拥挤。

#server side
sudo apt install -y apt-cacher-ng
sudo ss -lntp |grep apt-cacher-ng#client side
sudo apt -o Acquire::http::proxy="http://192.168.2.6:3142" update
echo 'Acquire::http::Proxy "http://192.168.2.6:3142";' | sudo tee /etc/apt/apt.conf.d/01acng

对于加速pypi, 可使用devpi缓存

#https://opensource.com/article/18/7/setting-devpi
sudo apt install python3-pip -y
sudo pip install devpi-server devpi-web
devpi-init --serverdir /bak/work/devpi
devpi-server --serverdir /bak/work/devpi --host 0.0.0.0pip install --trusted-host 192.168.2.6 -i http://192.168.2.6:3141/root/pypi/+simple/ httpieexport PIP_INDEX_URL=http://192.168.2.6:3141/root/pypi/+simple/
pip install httpieunset PIP_INDEX_URL
mkdir -p ~/.pip && cat > ~/.pip/pip.conf << EOF
[global]
index-url = http://192.168.2.6:3141/root/pypi/+simple/
[search]
index = http://192.168.2.6:3141/root/pypi/
[install]
trusted-host=192.168.2.6
disable-pip-version-check = true
timeout = 120
EOF

[可选] 创建虚机

例如，在win10上使用hypver-v创建一个7G(初始2G并动态分配即可)，40G硬件的ubuntu 20.04的名为o7k的虚机, 禁用check point减小硬盘使用：

Win10中搜索’hyper-v’打开’Turn Windows features on or off’将’Hyper-v’,　‘Virtual Machine Platform’, 'Windows Hypervior Platform’都打开
下载ubuntu 20.04 -　http://mirrors.aliyun.com/ubuntu-releases/20.04/ubuntu-20.04.2-live-server-amd64.iso
在’hyper-v manager’中点击’Create Virtual Machine’, 然后从’local　installtion source’中选择iso,　Disable ‘Secure Boot’, 内存7000M(初始2G并动态分配即可), 禁用检查点，其它设置默认
打开虚机里面的kvm嵌套虚拟化(参考：https://superuser.com/questions/1330271/configuring-kvm-inside-ubuntu-hyper-v-client), 先下载：https://raw.githubusercontent.com/MicrosoftDocs/Virtualization-Documentation/master/hyperv-tools/Nested/Enable-NestedVm.ps1　然后以管理员打开powershell, 先运行＂Set-ExecutionPolicy Bypass" 以允许在powershell中运行脚本，然后运行：.\Enable-NestedVm.ps1 o7k
[可选]安装docker用docker - https://hub.docker.com/editions/community/docker-ce-desktop-windows/，将devstack运行在docker容器中可参考：　https://github.com/slimjim2234/docker-devstack
[可选]安装完了，可以压缩:

# stop vm first
diskpart
select vdisk file="D:\images\o7k.vhdx"
compact vdisk
#compress it by winrar, then move it to D:\www\sftp, then we can download it by: sftp ftpuser@192.168.2.9X

将vhdx格式转化为其他格式

unrar x o7k.rar
qemu-img convert o7k.vhdx -O qcow2 o7k.qcow2 #kvm
qemu-img convert o7k.vhdx -O raw o7k.raw
qemu-img convert o7k.vhdx -O vmdk o7k.vmdk   #vmware
qemu-img convert o7k.vhdx -O vdi o7k.vdi     #virtualbox

例如，使用kvm来使用o7k.qcow2时，会遇到＂booting from hard disk ＂错误，那是因为hyper-v是默认使用UEFI来创建虚机的，那应该让KVM也支持UEFI:

sudo apt install ovmf && sudo systemctl restart libvirtd
ls /usr/share/OVMF/OVMF_CODE.fd
然后在virt-manager安前装在＂概况＂中将Firmware选择"UEFI x86_64: /usr/shar/OVMF/OVMF_CODE.fd"即可．

[可选]创建stack用户

ssh登录进虚机创建stack用户，此步可选, 注意，若添加了stack用户, 记得运行’sudo su - stack’之后做pip相关的设置:

echo 'ubuntu     ALL=(ALL) NOPASSWD:ALL' |sudo tee -a /etc/sudoers
echo 'precedence ::ffff:0:0/96 100' |sudo tee -a /etc/gai.conf
# don't add # when coping via vim
echo 'autocmd FileType * setlocal formatoptions-=c formatoptions-=r formatoptions-=o' |tee ~/.vimrcsudo useradd -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/stack
sudo su - stack && cd /opt/stack

[可选] 设置镜像

NOTE: 设置镜像太容易引发各种奇奇怪怪的包依赖问题，我本次未使用各种镜像．

使用deb镜像．NOTE：使用镜像，如果镜像不同步，可能会引入更多的包依赖问题．
我此次实际使用的是:cn.archive.ubuntu.com，你也可以使用：mirrors.cloud.tencent.com

sudo cp /etc/apt/sources.list /etc/apt/sources.list_bak
sudo sed -i s/archive.ubuntu.com/cn.archive.ubuntu.com/g /etc/apt/sources.list
sudo apt clean all
sudo apt update

使用pip镜像．NOTE: 使用镜像，如果镜像质量不好不同步，可能会引入更多的包依赖问题．
例如，我这次使用douban的pypi镜像时就遇到了下列包依赖的问题：

ERROR: Could not find a version that satisfies the requirement cliff===3.8.0 (from -c /bak/openstack/requirements/upper-constraints.txt (line 249)) (from versions: none)
ERROR: No matching distribution found for cliff===3.8.0 (from -c /bak/openstack/requirements/upper-constraints.txt (line 249))

所以我此次实际未使用镜像．附录１有本次使用douban镜像的步骤，仅供参考：

github（如git@github.com:openstack/nova.git）有可能被间歇性封锁, 建议尽量使用https协议避免使用git协议．也可考虑trystack的openstack代码镜像(http://git.trystack.cn ), 当然它的代码及时性就不好说了．真要开发肯定不能用这种，搭建测试环境暂时可以用．

运行devstack创建OpenStack环境

#NOTE: this can cause bastion unable to be SSHed
#juju deploy ubuntu openstack --series=focal --config hostname=openstack --constraints "mem=8G cores=2 root-disk=60G"
sudo mkdir -p /bak/openstack && sudo chown -R $(whoami) /bak/openstack && cd /bak/openstack
git clone https://github.com/openstack/devstack.git && cd devstack
#wget https://raw.githubusercontent.com/zhhuabj/exercise/master/o7k/local.conf
sudo bash -c 'cat > local.conf' <<EOF
[[local|localrc]]#OFFLINE=True
DEST=/bak/openstack
LIBVIRT_TYPE=qemu
mkdir -p $DEST
unset http_proxy
unset https_proxy
#sudo route del -net 10.0.1.0/24 gw 192.168.101.3 > /dev/null 2>&1
sudo apt -y install openvswitch-switch qemu-kvm libvirt-clients libvirt-daemon-system bridge-utils
#sudo virsh net-destroy default
sudo ovs-vsctl del-br br-data
sudo ovs-vsctl -- --may-exist add-br br-data
sudo ip addr add 172.16.1.1/24 dev br-data >/dev/null 2>&1 &
sudo chown -R $(whoami) /opt/stack/
#sudo ovs-vsctl -- --may-exist add-port br-data eth0 -- set interface eth0 type=internal# ENABLED_SERVICES=n-cpu,rabbit,neutron,q-agt
GIT_BASE="https://git.openstack.org"
#enable_plugin neutron-lbaas https://git.openstack.org/openstack/neutron-lbaas
ENABLED_SERVICES=rabbit,mysql,key,g-api,g-reg
ENABLED_SERVICES+=,n-api,n-crt,n-obj,n-cpu,n-cond,n-sch,n-cauth,n-novnc,placement-api,n-super-cond,n-cond-cell1
ENABLED_SERVICES+=,q-svc,q-agt,q-dhcp,q-l3,q-meta,neutron
disable_service q-metering
Q_USE_ROOTWRAP=True
Q_USE_ROOTWRAP_DAEMON=False
ENABLE_IDENTITY_V2=False
enable_service placement-api
heartbeat_timeout_threshold=7200
Q_ML2_TENANT_NETWORK_TYPE=flat,vlan,gre,vxlan
Q_ML2_PLUGIN_TYPE_DRIVERS=flat,vlan,gre,vxlan
# must use public
OVS_BRIDGE_MAPPINGS=public:br-data
PUBLIC_NETWORK_NAME=public# Neutron Agent configuration
Q_USE_NAMESPACE=True
Q_ALLOW_OVERLAPPING_IP=True
# Neutron ML2 configuration
Q_PLUGIN=ml2
#Q_DVR_MODE=dvr_snat
Q_DVR_MODE=legacy
# VLAN
ENABLE_TENANT_VLANS=True
ML2_VLAN_RANGES=public:1000:2999
# GRE
ENABLE_TENANT_TUNNELS=True
TENANT_TUNNEL_RANGE=1000:2000
Q_ML2_PLUGIN_VXLAN_TYPE_OPTIONS=(vni_ranges=1000:2000)# Avanced ML2 configuration
Q_AGENT=openvswitch
Q_ML2_PLUGIN_MECHANISM_DRIVERS=openvswitch,linuxbridge,l2population# OVN
#enable_plugin networking-ovn http://git.openstack.org/openstack/networking-ovn
#enable_service ovn-northd
#enable_service ovn-controller
# We have to disable the neutron L2 agent. OVN does not use the L2 agent.
#disable_service q-agt# Swift
#ENABLED_SERVICES+=,s-proxy,s-object,s-container,s-account
#sudo apt -y install libapache2-mod-wsgi
VOLUME_BACKING_FILE_SIZE=500M
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data/swift
NOVA_VNC_ENABLED=True# Ceph
sudo rm -rf /var/lib/ceph/mon/ceph-`hostname`
#ENABLED_SERVICES+=,ceph
CEPH_LOOPBACK_DISK_SIZE=3G
CEPH_CONF=/etc/ceph/ceph.conf
CEPH_REPLICAS=1# Cinder - Block Device Service
#ENABLED_SERVICES+=,cinder,c-api,c-vol,c-sch,c-bak
CINDER_DRIVER=ceph
CINDER_CEPH_USER=cinder
CINDER_CEPH_POOL=volumes
CINDER_CEPH_UUID=6d52eb95-12f3-47e3-9eb9-0c1fe4142426
CINDER_BAK_CEPH_POOL=backeups
CINDER_BAK_CEPH_USER=cind-backeups
CINDER_ENABLED_BACKENDS=ceph,lvm
GLANCE_CEPH_USER=glance
GLANCE_CEPH_POOL=images
NOVA_CEPH_POOL=vmzHOST_IP=172.16.1.1
SERVICE_HOST=$HOST_IP
MYSQL_HOST=$SERVICE_HOST
#QPID_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
Q_HOST=$SERVICE_HOSTFIXED_RANGE=10.0.1.0/24
#113 -- 118, http://jodies.de/ipcalc
FLOATING_RANGE=192.168.101.0/24
Q_FLOATING_ALLOCATION_POOL=start=192.168.101.3,end=192.168.101.100
PUBLIC_NETWORK_GATEWAY=192.168.101.1
NETWORK_GATEWAY=10.0.1.1
PUBLIC_BRIDGE=br-data
# sudo ovs-vsctl add-port br-ex eth0
OVS_PHYSICAL_BRIDGE=br-data
# https://review.openstack.org/#/c/87987/
IP_VERSION=4
DATABASE_USER=root
DATABASE_PASSWORD=password
ADMIN_PASSWORD=password
SERVICE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_TOKEN=ADMIN
LOGFILE=$DEST/logs/stack.log
ENABLE_DEBUG_LOG_LEVEL=False
SYSLOG=False
VERBOSE=True
SCREEN_LOGDIR=$DEST/logs
LOG_COLOR=False
Q_USE_DEBUG_COMMAND=False
KEYSTONE_DEPLOY=mod_wsgi
USE_SSL=False
disable_service tls-proxy
USE_PYTHON3=True
DOWNLOAD_DEFAULT_IMAGES=False
IMAGE_URLS="http://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img"
#IPSEC_PACKAGE=strongswan# don't use trystack git mirror, but just use noVNC from trystack
#GIT_BASE=http://git.trystack.cn
# git call failed: [git clone' https://github.com/novnc/noVNC.git
NOVNC_REPO=http://git.trystack.cn/kanaka/noVNC.git
#SPICE_REPO=http://git.trystack.cn/git/spice/sice-html5.gidisable_service etcd3
disable_service memory_tracker#devstack NET_ID 'Failure creating NET_ID for private
#NEUTRON_CREATE_INITIAL_NETWORKS=False#HORIZON_REPO=https://github.com/openstack/horizon
#HORIZON_BRANCH=stable/grizzly
#CINDER_REPO=https://github.com/openstack/cinder.git
#CINDER_BRANCH=stable/grizzly
#CINDERCLIENT_REPO=https://github.com/openstack/python-cinderclient.git
#CINDERCLIENT_BRANCH=stable/grizzly
#NOVA_REPO=https://github.com/openstack/nova.git
#NOVA_BRANCH=stable/grizzly
#NOVACLIENT_REPO=https://github.com/openstack/python-novaclient.git
#NOVACLIENT_BRANCH=stable/grizzly
#GLANCE_REPO=https://github.com/openstack/glance.git
#GLANCE_BRANCH=stable/grizzly
#GLANCECLIENT_REPO=https://github.com/openstack/python-glanceclient.git
#KEYSTONE_REPO=https://github.com/openstack/keystone.git
#KEYSTONE_BRANCH=stable/grizzly
#KEYSTONECLIENT_REPO=https://github.com/openstack/python-keystoneclient.git
#KEYSTONECLIENT_BRANCH=stable/grizzly
#NEUTRON_REPO=https://github.com/openstack/neutron.git
#NEUTRON_BRANCH=stable/grizzly
#NEUTRONCLIENT_REPO=https://github.com/openstack/python-neutronclient.git
#NEUTRONCLIENT_BRANCH=stable/grizzly
#CEILOMETER_REPO=git://git.openstack.org/openstack/ceilometer.git
#CEILOMETERCLIENT_REPO=git://git.openstack.org/openstack/python-ceilometerclient.git
#HEAT_REPO=git://git.openstack.org/openstack/heat.git
#HEAT_BRANCH=master
#SWIFT_REPO=git://git.openstack.org/openstack/swift.git
#SWIFT_BRANCH=master
#OPENSTACKCLIENT_REPO=git://git.openstack.org/openstack/python-openstackclient.git
#OPENSTACKCLIENT_BRANCH=master
#SOLUM_REPO=git://git.openstack.org/stackforge/solum.git
#SOLUM_BRANCH=master# Enable Ceilometer
#enable_service ceilometer-acompute
#enable_service ceilometer-acentral
#enable_service ceilometer-anotification
#enable_service ceilometer-collector
#enable_service ceilometer-api
#enable_service ceilometer-alarm-notifier
#enable_service ceilometer-alarm-evaluator
#CEILOMETER_PIPELINE_INTERVAL=60# Enable Solum
#enable_service solum
#enable_service solum-api
#enable_service solum-build-api
#enable_service solum-deployer
#enable_service solum-conductor
#enable_service solum-worker
#SOLUM_INSTALL_CEDARISH=True# Enable Marconi
#disable_service marconi-server
#enable_service zaqar-server#https://wiki.openstack.org/wiki/Docker
#export INSTALLDIR=/bak/openstack && cd $INSTALLDIR/nova-docker
#git clone https://git.openstack.org/stackforge/nova-docker
#./contrib/devstack/prepare_devstack.sh
#VIRT_DRIVER=docker
#DEFAULT_IMAGE_NAME=cirros
#NON_STANDARD_REQS=1
#IMAGE_URLS=""#Enable tacker
#TACKER_BRANCH=master
#enable_plugin heat https://git.openstack.org/openstack/heat $TACKER_BRANCH
#enable_plugin networking-sfc https://git.openstack.org/openstack/networking-sfc $TACKER_BRANCH
#enable_plugin barbican https://git.openstack.org/openstack/barbican $TACKER_BRANCH
#enable_plugin mistral https://git.openstack.org/openstack/mistral $TACKER_BRANCH
#enable_plugin tacker https://git.openstack.org/openstack/tacker $TACKER_BRANCH
EOF

然后使用＂./stack.sh＂运行它，注意非常重要的一点，如果想要重新运行避免出莫名其妙的错误的话，必须：

#re-run
sudo systemctl stop "devstack@*"
./clean.sh
./stack.sh

Use it

source accrc/admin/admin
$ env |grep OS_
OS_PROJECT_DOMAIN_ID=default
OS_CACERT=
OS_AUTH_URL=http://172.16.1.1/identity
OS_USER_DOMAIN_ID=default
OS_USERNAME=admin
OS_PROJECT_NAME=admin
OS_PASSWORD=passwordsource <(openstack complete)
openstack complete |sudo tee /etc/bash_completion.d/openstack.bash_completion > /dev/nullsystemctl -a |grep devstack
sudo tail -f /var/log/syslog
sudo journalctl -f --unit devstack@n-cpu.service# only need to create network/subnet staffs when NEUTRON_CREATE_INITIAL_NETWORKS=true
openstack router create provider-router
openstack network create private
openstack subnet create --subnet-range 10.0.1.0/24 --network private --allocation-pool start=10.0.1.50,end=10.0.1.100 --gateway 10.0.1.1 private-subnet
openstack network create --provider-network-type flat --external --share --provider-physical-network physnet1 ext_net
openstack subnet create --allocation-pool start=192.168.101.3,end=192.168.101.100 --subnet-range 192.168.101.0/24 --dhcp --gateway 192.168.101.1 --ip-version 4 --network ext_net ext_net_subnet
#openstack subnet set ext_net_subnet --dhcp  #just simulate ext_net so use dhcp
openstack router add subnet provider-router private-subnet
openstack router set --external-gateway ext_net provider-routercat << EOF > user-data
#cloud-config
user: ubuntu
password: password
chpasswd: { expire: False }
EOF
# in case you don't have a key
#ssh-keygen -q -N ""
openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey
openstack --debug server create --image cirros-0.5.2-x86_64-disk --flavor m1.tiny --key-name mykey --nic net-id=$(openstack network list | awk '/private/ {print $2}') --user-data ./user-data --config-drive true i1#add FIP
public_network=$(openstack network show public -f value -c id)
fip=$(openstack floating ip create $public_network -f value -c floating_ip_address)
fix_ip=$(openstack server list |grep 'private=' |awk -F '=' '{print $2}' |awk '{print $1}')
openstack floating ip set $fip --fixed-ip-address $fix_ip --port $(openstack port list --fixed-ip ip-address=$fix_ip -c id -f value)#add SG rules
sg_id=$(openstack security group list --project=$(openstack server show i1 -f value -c project_id) |grep default |awk '{print $2}')
openstack security group rule create --protocol tcp --dst-port 22 $sg_id
openstack security group rule create --protocol icmp $sg_id
sudo ip netns exec qrouter-6aef358e-7cbe-4d77-af70-ed13fd11f621 ping $fix_ip
openstack console log show i1
#we just simulate fip so we should ping fip from qg-xxx interface
sudo ip netns exec qrouter-6aef358e-7cbe-4d77-af70-ed13fd11f621 ping $fip

Debug it

sudo su - stack
cd /bak/openstack/devstack
sudo systemctl --failed
sudo systemctl list-units |grep devstack |grep failed |awk '{print $2}'
sudo systemctl status devstack@*
source accrc/admin/admin
openstack server list# if above doesn't work, we may need to rerun devstack
sudo systemctl stop devstack@*
./stack.sh
openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey
# 有时创建虚机失败时，换个名字如i2就好了　
openstack --debug server create --image cirros-0.5.2-x86_64-disk --flavor m1.tiny --key-name mykey --nic net-id=$(openstack network list | awk '/private/ {print $2}') i1
# if fails, pls check
sudo systemctl --failed
sudo tail -f /var/log/syslog
sudo systemctl -a |grep devstack@
sudo systemctl restart devstack@n-cpu.service
sudo journalctl -f --unit devstack@n-cpu.service
see - https://docs.openstack.org/devstack/latest/systemd.html

附录1 - 使用pip镜像

pypi.douban.com的pip镜像质量不高，同步不及时，导致我在此次使用过程中遇到了下列包依赖问题．

ERROR: Could not find a version that satisfies the requirement cliff===3.8.0 (from -c /bak/openstack/requirements/upper-constraints.txt (line 249)) (from versions: none)
ERROR: No matching distribution found for cliff===3.8.0 (from -c /bak/openstack/requirements/upper-constraints.txt (line 249))

去掉它就好了，下面只是记录使用douban镜像的过程，仅作参考：
NOTE: 一定要使用pip镜像时也可以试试清华的（pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple　）

当前用户或stack用户添加：

mkdir ~/.pip/
cat << EOF > ~/.pip/pip.conf
[global]
index-url = http://pypi.douban.com/simple
[install]
trusted-host=pypi.douban.com
disable-pip-version-check = true
timeout = 120
EOF
此时，"sudo pip3 install -r requirements.txt＂还是未生效，那是因为需要继续在root用户中也添加：
sudo mkdir /root/.pip
sudo bash -c 'cat > /root/.pip/pip.conf' << EOF
[global]
index-url = http://pypi.douban.com/simple
[install]
trusted-host=pypi.douban.com
disable-pip-version-check = true
timeout = 120
EOF
或者使用：
[global]
trusted-host=mirrors.aliyun.com
index-url = http://mirrors.aliyun.com/pypi/simple
disable-pip-version-check = true
timeout = 120若不使用pip镜像，例如，运行下列命令时，
sudo -H LC_ALL=en_US.UTF-8 http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite python3.8 -m pip install -c /bak/openstack/requirements/upper-constraints.txt 'glance-store[cinder]!=0.29.0'
会报这种错误：
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
可以局部通过如下添加＂--index-url https://pypi.douban.com/simple＂解决：
sudo -H LC_ALL=en_US.UTF-8 http_proxy= https_proxy= no_proxy= PIP_FIND_LINKS= SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite python3.8 -m pip install -c /bak/openstack/requirements/upper-constraints.txt 'glance-store[cinder]!=0.29.0' --index-url https://pypi.douban.com/simple
对于小包可以通过添加超时时间解决（pip --default-timeout=100 install -U pip）

附录2 - 其他可能的问题

1, 当将devstack中的OFFLINE设置为True后重新运行./stack.sh报下列错：
apache2.service: Failed with result 'start-limit-hit'
Soluation:
从这个网页找到了答案：https://www.hiroom2.com/2017/02/18/linux-systemd-s-start-request-repeated-too-quickly-for-xxx-service/
修改文件（/lib/systemd/system/apache2.service）添加下设置禁用掉systemd对重启服务次数的限制．
StartLimitIntervalSec=0
StartLimitBurst=0
然后运行
sudo systemctl daemon-reload
sudo systemctl reset-failed apache2.service2, 运行＂cd /bak/openstack/horizon && /usr/bin/python3.8 manage.py compilemessages＂报下列错：
ModuleNotFoundError: No module named 'django.utils.log.NullHandler'; 'django.utils.log' is not a package
Solution:
sed -i "s/django.utils.log.NullHandler/logging.NullHandler/g" ./openstack_dashboard/local/local_settings.py3, 运行＂sudo ebtables -t broute -L＂报下列错：
table `broute' is incompatible, use 'nft' tool
Solution:
Try Switching to iptable-legacy
sudo apt update && sudo apt upgrade
sudo apt -y install iptables arptables ebtables
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy || true
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy || true
sudo update-alternatives --set arptables /usr/sbin/arptables-legacy || true
sudo update-alternatives --set ebtables /usr/sbin/ebtables-legacy || true

20210915更新　- 运行devstack镜像注意点

#注意点
1, 实验用的虚机镜像是UEFI格式的，所以导入虚机镜像时记得使用UEFI。
2, 虚机启动后没有自动获得IP的问题，那是因为：a, 物理机libvirt的default网络若使用了默认的192.168.122.0/24网段，虚机内部也安装了libvirt会用这个默认网段。虚机内部使用virsh net-edit default修改一个不同网段并重启虚机即可。b, 若虚机内的网卡名不叫eth0, 如enp1s0.需修改/etc/netplan/00-installer-config.yaml将eth0改为enps1s0, 然后运行sudo netplan apply即可获得IP
3, 最好devstack运行成功之后禁用它的重新开机后的自动重启。因为devstack机器重启之后总是会遇到各种问题，索性每次重运行一次 sudo systemctl -a |grep devstack@ |xargs sudo systemctl disablesudo systemctl -a#重新运行, OFFLINE=true模式不work, 仍应确保使用unstack.sh而不是clean.sh
sudo su - stack
cd /bak/openstack/devstack
#./clean.sh
./unstack.sh
./stack.sh#运行成功之后记得及时disable所有服务避免下次重启机器时再各种问题
sudo systemctl -a |grep devstack@ |xargs sudo systemctl disable#验证
sudo systemctl --failed
sudo systemctl status devstack@*
source accrc/admin/admin
openstack server list#使用, 有时创建虚机会失败，换个名字如i2就好了　
openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey
openstack --debug server create --image cirros-0.5.2-x86_64-disk --flavor m1.tiny --key-name mykey --nic net-id=$(openstack network list | awk '/private/ {print $2}') i1#访问虚机
PROJECT_ID=$(openstack project show --domain default admin -f value -c id)
SG_ID=$(openstack security group list --project ${PROJECT_ID} | awk '/default/ {print $2}')
openstack security group rule list $SG_ID
openstack security group rule create $SG_ID --ingress --ethertype IPv4 --protocol icmp --remote-ip 0.0.0.0/0
openstack security group rule create $SG_ID --ingress --ethertype IPv4 --protocol tcp --dst-port 22:22 --remote-ip 0.0.0.0/0
#you may fail to ssh cirros becuase dropbear key and openssh key are incompatible, change non-cirros will succeed
#sudo ip netns exec qrouter-b4a099e0-d0db-4ec8-8f3c-9d83e11d3488 ssh cirros@10.0.1.20 -v#浮动IP
# eg: vm=10.0.1.20
public_network=$(openstack network show public -f value -c id)
fip=$(openstack floating ip create $public_network -f value -c floating_ip_address)
openstack floating ip set $fip --fixed-ip-address 10.0.1.20 --port $(openstack port list --fixed-ip ip-address=10.0.1.20 -c id -f value)#调试
sudo systemctl --failed
sudo tail -f /var/log/syslog
sudo systemctl -a |grep devstack@
sudo systemctl restart devstack@n-cpu.service
sudo journalctl -f --unit devstack@n-cpu.service