温故OpenStack中的测试(by Joshua)

沿用tox调用virtualenv自动创建的虚拟环境(virtualenv -p python3.5 .tox/py35)

source .tox/py35/bin/activate
sudo pip install --upgrade -r requirements.txt
sudo pip install --upgrade -r test-requirements.txt

使用unittest和nose运行测试。nose是对unittest的扩展，使得python的测试更加简单，nose自动发现测试代码并执行，nose提供了大量的插件，比如覆盖报表等。

python -m unittest -v unit_tests.test_neutron_utils.TestNeutronUtils.test_get_packages_ovs_newton
.tox/py35/bin/python nosetests -v unit_tests/test_neutron_utils.py:TestNeutronUtils.test_get_packages_ovs_newton

注意：上面采用nosetests运行时会报错，因为我们的测试采用了python3, 所以需要在安装了python3-nose之后（sudo apt-get install python3-nose python3-mock）再采用下列三种方式之一运行：

nosetests3 -v unit_tests/test_neutron_utils.py:TestNeutronUtils.test_restart_map_ovs_odl
/bak/work/charms/neutron-gateway/.tox/py35/bin/python /usr/local/bin/nosetests -v unit_tests/test_neutron_utils.py:TestNeutronUtils.test_get_packages_ovs_newton
python -m nose unit_tests/test_neutron_utils.py:TestNeutronUtils.test_restart_map_ovs_odl

但实际上仍然找不找nose模块，那是因为nose与virtualenv结合地不大好，在这个网页找着了答案(https://stackoverflow.com/questions/864956/problems-using-nose-in-a-virtualenv) - You need to have a copy of nose installed in the virtual environment. In order to force installation of nose into the virtualenv, even though it is already installed in the global site-packages, run pip install with the -I flag: pip install nose -I

上面使用unittest与nose运行测试的方式只是将结果输出到stdout，不便于分析。所以可以使用python-subunit模块来运行测试，并将测试结果通过subunit协议输出到文件中便于日后分析。因为subunit是基于二进制的不便于人眼看，所以可使用subunit2pyunit工具将其人类可读化。

python -m subunit.run discover |subunit2pyunit
python -m subunit.run discover -t ./ ./unit_tests |subunit2pyunit
python -m subunit.run unit_tests.test_neutron_utils.TestNeutronUtils. |subunit2pyunit

在大型应用中分析测试结果很重要，testrepository可以调用subunit来用python-subunit模块来运行测试，并将测试结果通过subunit协议输出到文件中，然后testrepository在些基础上有更多的分析，如分析哪些用例运行的时间最长，如显示失败的用例，如仅运行上次运行失败的用例。

testr init
testr run
testr run --parallel
$ cat .testr.conf
[DEFAULT]
test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \${PYTHON:-python} -m subunit.run discover -t ./ ./unit_tests $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list

tox用于创建虚拟python环境，也可以集成上面的testrepository(commands = ostestr {posargs})

$ cat tox.ini
[tox]
envlist = pep8,py27,py35
skipsdist = True[testenv]
setenv = VIRTUAL_ENV={envdir}PYTHONHASHSEED=0CHARM_DIR={envdir}AMULET_SETUP_TIMEOUT=5400
install_command =pip install --allow-unverified python-apt {opts} {packages}
commands = ostestr {posargs}
whitelist_externals = juju
passenv = HOME TERM AMULET_* CS_API_*[testenv:py27]
basepython = python2.7
deps = -r{toxinidir}/requirements.txt-r{toxinidir}/test-requirements.txt
commands = /bin/true[testenv:py35]
basepython = python3.5
deps = -r{toxinidir}/requirements.txt-r{toxinidir}/test-requirements.txt[testenv:pep8]
basepython = python2.7
deps = -r{toxinidir}/requirements.txt-r{toxinidir}/test-requirements.txt
commands = flake8 {posargs} hooks unit_tests tests actions libcharm-proof[flake8]
ignore = E402,E226
exclude = */helpers

pydev使用virtualenv中的py35

在eclipse的"Preferences -> Pydev -> Interpreters -> Python Interpreters"菜单中定义python35=/bak/work/charms/neutron-gateway/.tox/py35/bin/python,然后在工程上点右键从"Properties -> Pydev - Interpreter/Grammar"定义使用python35。注意，需要将/bak/work/charms/neutron-gateway/.tox/py35/lib/python3.5/site-packages也选到环境变量中，否则后面会报ImportError: No module named 'mock。
为一个测试类定义"Python unitest"类型的"Debug Configurations", 也在其Interpreter选项卡中定义使用python35 (结果：eclipse似乎有bug，此处选择了python35后无法保存)
所以无法成功，似乎是pydev与python3协作不大好。最后还是pudb好使(sudo pip install pudb, import pudb; pdb.set_trace())
py27下的测试运行方法(如openstack), charm似乎只能用py36 (tox -r -epy36 && tox -e py36).

cat tox.ini
tox -r -epy27
tox -e py27,pep8
tox -e py27 neutron.tests.unit.agent.linux.test_keepalived
tox -e py27 neutron.tests.unit.agent.linux.test_keepalived.KeepalivedInstanceTestCase.test_remove_addresses_by_interface
mkdir /opt/stack && sudo chown -R hua /opt/stack
tox -e functional neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_keepalived_configuration
tox -e dsvm-fullstack
source .tox/functional/bin/activate
.tox/functional/bin/pip install -r requirements.txt
.tox/functional/bin/pip install -r test-requirements.txt
.tox/functional/bin/pip install -r neutron/tests/functional/requirements.txt
.tox/functional/bin/pip freeze |grep neutron
.tox/functional/bin/pip install 'neutron-lib==1.13.0'
OS_SUDO_TESTING=True .tox/functional/bin/python -m unittest -v neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_keepalived_configuration

有时如mitaka已经eol了, 例如它的origain-stable-mitaka这个分支都没有了, 这会导致运行’tox -r -epy27 '不成功, 那么可以手工执行:

virtualenv venv_mitaka
. venv_mitaka/bin/activate# pg_config executable not found
# Error: could not determine PostgreSQL version from '10.5'
sudo apt install python-setuptools python-dev libpython-dev libssl-dev python-pip libmysqlclient-dev libxml2-dev libxslt-dev libxslt1-dev libpq-dev git git-review libffi-dev gettext graphviz libjpeg-dev zlib1g-dev build-essential python-nose python-mock libssl1.0
sudo apt install python3.6 python3.6-dev python3-pip python3-dev python3-nose python3-mock
#sudo pip install --upgrade setuptools
#sudo pip3 install --upgrade setuptools
#sudo pip install --upgrade --force-reinstall pip virtualenv# Failed to install Cryptography - sudo apt install libssl1.0
# No module named dulwich - ./venv/bin/pip install dulwich
# unittest has no attribute 'virt' -
# ./venv_ocata/bin/pip install -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/ocata .
./venv_mitaka/bin/pip install -c https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=mitaka-eol .
./venv_mitaka/bin/pip install -r requirements.txt
./venv_mitaka/bin/pip install -r test-requirements.txtfind . -name "*.pyc" -exec rm -rf {} \;git clone https://github.com/openstack/oslo.cache.git
cd oslo.cache && git checkout -b mitaka mitaka-eol
../nova/venv/bin/pip install -r requirements.txt
../nova/venv/bin/pip install -r test-requirements.txt# dogpile can't cannot import name threading - venv/bin/pip uninstall dogpile.cache dogpile && venv/bin/pip install dogpile.cache
venv_mitaka/bin/python -m subunit.run discover ./nova/tests/unit/compute |subunit2pyunit
venv_mitaka/bin/python -m subunit.run nova.tests.unit.compute.test_compute |subunit2pyunit
venv_mitaka/bin/python -m subunit.run nova.tests.unit.compute.test_compute.ComputeAPITestCase.test_attach_volume |subunit2pyunit
venv_mitaka/bin/python -m unittest -v nova.tests.unit.compute.test_compute  #make sure the package nova.tests.virt existsvenv_mitaka/bin/python -m unittest -v nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_check_can_live_migrate_dest_all_pass_with_over_commit#venv/bin/pep8
venv/bin/flake8

对于horizion的测试:

https://docs.openstack.org/horizon/latest/contributor/testing.html
tox
tox -e py27 -- openstack_dashboard.test.views:DashboardViewsTest.test_urls_ngdetails

charm的unit test:

# charm unit test
sudo pip install virtualenv
sudo pip install --upgrade pip
proxychains tox -r -epy36
source .tox/py36/bin/activate
python -m subunit.run discover -t ./ ./unit_tests |subunit2pyunit
python -m subunit.run unit_tests.test_neutron_ovs_context.L3AgentContextTest |subunit2pyunit
python -m subunit.run unit_tests.test_neutron_ovs_context.L3AgentContextTest.test_dvr_enabled |subunit2pyunit
OS_SUDO_TESTING=True .tox/py36/bin/python -m unittest -v unit_tests.test_neutron_ovs_context.L3AgentContextTest.test_dvr_enabled

20200714更新

今天，发现上面的subunit的方法不work了，　例如运行下面三行报这个错＂TypeError: test_data_port_name() missing 1 required positional argument: ‘config’＂

tox -r -epy38
source .tox/py38/bin/activate
python -m subunit.run unit_tests.test_neutron_ovs_context.OVSPluginContextTest.test_data_port_name

估计是现在全面移到python3了，subunit和python3搭配不大好吧，　试了一下，　下面两种方法还可以用：

OS_SUDO_TESTING=True .tox/py3/bin/python -m unittest -v subunit.run unit_tests.test_neutron_ovs_context.OVSPluginContextTest.test_data_port_name
nosetests3 unit_tests/test_neutron_ovs_context.py:OVSPluginContextTest.test_data_port_name

20201026更新

#https://review.opendev.org/#/c/749175/
cd nova
tox
#tox -r -epy38
#source .tox/py38/bin/activate
tox -epy38 nova.tests.unit.pci.test_stats.PciDeviceStatsWithTagsTestCase.test_update_device
.tox/py38/bin/python -m unittest nova.tests.unit.pci.test_stats.PciDeviceStatsWithTagsTestCase.test_update_devicetox -e functional nova.tests.functional.libvirt.test_pci_sriov_servers.SRIOVServersTest.test_create_server_after_change_in_nonsriov_pf_to_sriov_pf
.tox/functional/bin/python -m unittest -v nova.tests.functional.libvirt.test_pci_sriov_servers.SRIOVServersTest.test_create_server_after_change_in_nonsriov_pf_to_sriov_pf#NOTE: can use rpdb to debug nova/tests/functional/libvirt/test_pci_sriov_servers.py
.tox/functional/bin/pip3 install rpdb
import rpdb;rpdb.set_trace()
nc 127.0.0.1 4444python3 -m flake8

unit test怎么写

下列代码是初始为https://bugs.launchpad.net/charm-octavia/+bug/1915512　写的，它是错的因为在删除一个unit时，leader马上会运行下列代码仍然会将正在删除但还未删除的unit弄进去所以是错的。

+        running_units.append(ch_core.hookenv.local_unit().replace('/', '-'))
+        for u in ch_core.hookenv.iter_units_for_relation_name('cluster'):
+            running_units.append(u.unit.replace('/', '-'))

但是这个代码刚好可以演示unit test怎么写。

From 02d8c5185afc6aebfa03d58aa1fe90be4b0d200f Mon Sep 17 00:00:00 2001
From: xxxx
Date: Fri, 23 Apr 2021 18:10:32 +0800
Subject: [PATCH] Delete hm prt on unit removalJuju has no pre hook to clean up hm port resources first before removing
one octavia unit. This patch will do it in another way, the leader unit finds
all running units by querying 'cluster' relations, then finds all mapping
between IPv6 management address and unit name by querying DB, finally
deletes hm ports for missing units.Closes-Bug: 1915512
Change-Id: I88c61b8d2d0b573df7df071ed7978e83b6803c5c
---src/lib/charm/openstack/api_crud.py           | 26 ++++++++++++++++++-src/reactive/octavia_handlers.py              | 26 ++++++++++++++++++-.../test_lib_charm_openstack_api_crud.py      | 25 +++++++++++++++---unit_tests/test_octavia_handlers.py           | 16 +++++++-----4 files changed, 81 insertions(+), 12 deletions(-)diff --git a/src/lib/charm/openstack/api_crud.py b/src/lib/charm/openstack/api_crud.py
index 81c8b31..a0209b8 100644
--- a/src/lib/charm/openstack/api_crud.py
+++ b/src/lib/charm/openstack/api_crud.py
@@ -251,6 +251,26 @@ def lookup_hm_port(nc, local_unit_name):return+def delete_hm_port(identity_service, local_unit_name):
+    """Delete port object for Octavia hm port for local unit.
+
+    :param nc: Neutron Client object
+    :type nc: neutron_client.Client
+    :param local_unit_name: Name of juju unit, used to build tag name for port
+    :type local_unit_name: str
+    :returns: None
+    :raises: DuplicateResource or any exceptions raised by Keystone and Neutron
+             clients.
+    """
+    session = session_from_identity_service(identity_service)
+    try:
+        nc = init_neutron_client(session)
+        port = lookup_hm_port(nc, local_unit_name)
+        nc.delete_port(port['id'])
+    except NEUTRON_TEMP_EXCS as e:
+        raise APIUnavailable('neutron', 'ports', e)
+
+def get_hm_port(identity_service, local_unit_name, local_unit_address,host_id=None):"""Get or create a per unit Neutron port for Octavia Health Manager.
@@ -507,12 +527,16 @@ def get_port_ips(identity_service):except NEUTRON_TEMP_EXCS as e:raise APIUnavailable('neutron', 'ports', e)+    neutron_ip_unit_map = dict()neutron_ip_list = []for port in resp['ports']:for ip_info in port['fixed_ips']:neutron_ip_list.append(ip_info['ip_address'])
+            unitname = port['name'].replace(
+                'octavia-health-manager-', '').replace('-listen-port', '')
+            neutron_ip_unit_map[unitname] = ip_info['ip_address']-    return neutron_ip_list
+    return neutron_ip_list, neutron_ip_unit_mapdef get_mgmt_network(identity_service, create=True):
diff --git a/src/reactive/octavia_handlers.py b/src/reactive/octavia_handlers.py
index c1f57d2..cbb7d67 100644
--- a/src/reactive/octavia_handlers.py
+++ b/src/reactive/octavia_handlers.py
@@ -179,7 +179,9 @@ def update_controller_ip_port_list():leader_ip_list = leadership.leader_get('controller-ip-port-list') or []try:
-        neutron_ip_list = sorted(api_crud.get_port_ips(identity_service))
+        neutron_ip_list, neutron_ip_unit_map = api_crud.get_port_ips(
+            identity_service)
+        neutron_ip_list = sorted(neutron_ip_list)except api_crud.APIUnavailable as e:ch_core.hookenv.log('Neutron API not available yet, deferring ''port discovery. ("{}")'
@@ -187,6 +189,28 @@ def update_controller_ip_port_list():level=ch_core.hookenv.DEBUG)returnif neutron_ip_list != sorted(leader_ip_list):
+        # delete hm port which may be caused by 'juju remove-unit <octavia>'
+        missing_units = []
+        running_units = []
+        db_units = neutron_ip_unit_map.keys()
+        running_units.append(ch_core.hookenv.local_unit().replace('/', '-'))
+        for u in ch_core.hookenv.iter_units_for_relation_name('cluster'):
+            running_units.append(u.unit.replace('/', '-'))
+        for unit_name in db_units:
+            if unit_name not in running_units:
+                missing_units.append(unit_name)
+        for unit_name in missing_units:
+            neutron_ip_list.remove(neutron_ip_unit_map.get(unit_name))
+            try:
+                ch_core.hookenv.log('deleting hm port for missing '
+                                    'unit {}'.format(unit_name))
+                api_crud.delete_hm_port(identity_service, unit_name)
+            except api_crud.APIUnavailable as e:
+                ch_core.hookenv.log('Neutron API not available yet, deferring '
+                                    'port discovery. ("{}")'
+                                    .format(e),
+                                    level=ch_core.hookenv.DEBUG)
+                returnleadership.leader_set({'controller-ip-port-list': json.dumps(neutron_ip_list)})diff --git a/unit_tests/test_lib_charm_openstack_api_crud.py b/unit_tests/test_lib_charm_openstack_api_crud.py
index cbc5776..5ca2628 100644
--- a/unit_tests/test_lib_charm_openstack_api_crud.py
+++ b/unit_tests/test_lib_charm_openstack_api_crud.py
@@ -247,6 +247,19 @@ class TestAPICrud(test_utils.PatchHelper):nc.update_port.assert_called_with('fake-port-uuid',{'port': {'admin_state_up': True}})+    def test_delete_hm_port(self):
+        self.patch_object(api_crud, 'session_from_identity_service')
+        self.patch_object(api_crud, 'init_neutron_client')
+        identity_service = mock.MagicMock()
+        nc = mock.MagicMock()
+        self.init_neutron_client.return_value = nc
+        nc.list_ports.return_value = {'ports': [{'id': 'fake-port-uuid'}]}
+        api_crud.delete_hm_port(identity_service, 'fake-unit-name')
+        self.init_neutron_client.assert_called_once_with(
+            self.session_from_identity_service())
+        nc.list_ports.asssert_called_with(tags='charm-octavia-fake-unit-name')
+        nc.delete_port.assert_called_with('fake-port-uuid')
+def test_setup_hm_port(self):self.patch('subprocess.check_output', 'check_output')self.patch('subprocess.check_call', 'check_call')
@@ -305,14 +318,18 @@ class TestAPICrud(test_utils.PatchHelper):self.init_neutron_client.return_value = ncnc.list_ports.return_value = {'ports': [
-                {'fixed_ips': [{'ip_address': '2001:db8:42::42'}]},
-                {'fixed_ips': [{'ip_address': '2001:db8:42::51'}]},
+                {'name': 'octavia-health-manager-lb-0-listen-port',
+                 'fixed_ips': [{'ip_address': '2001:db8:42::1'}]},
+                {'name': 'octavia-health-manager-lb-1-listen-port',
+                 'fixed_ips': [{'ip_address': '2001:db8:42::2'}]},],}identity_service = mock.MagicMock()
+        fake_ip_list = ['2001:db8:42::1', '2001:db8:42::2']
+        fake_ip_unit_map = {'lb-0': '2001:db8:42::1', 'lb-1': '2001:db8:42::2'}
+        fake_return_value = (fake_ip_list, fake_ip_unit_map)self.assertEquals(api_crud.get_port_ips(identity_service),
-                          ['2001:db8:42::42',
-                           '2001:db8:42::51'])
+                          fake_return_value)self.init_neutron_client.assert_called_once_with(self.session_from_identity_service())diff --git a/unit_tests/test_octavia_handlers.py b/unit_tests/test_octavia_handlers.py
index 4e3a3c5..8b2560d 100644
--- a/unit_tests/test_octavia_handlers.py
+++ b/unit_tests/test_octavia_handlers.py
@@ -171,16 +171,20 @@ class TestOctaviaHandlers(test_utils.PatchHelper):self.patch('charms.leadership.leader_set', 'leader_set')self.patch('charms.leadership.leader_get', 'leader_get')self.patch_object(handlers.api_crud, 'get_port_ips')
-        self.get_port_ips.return_value = [
-            '2001:db8:42::42',
-            '2001:db8:42::51',
-        ]
+        fake_ip_list = ['2001:db8:42::1', '2001:db8:42::2']
+        fake_ip_unit_map = {'lb-0': '2001:db8:42::1', 'lb-1': '2001:db8:42::2'}
+        self.get_port_ips.return_value = [fake_ip_list, fake_ip_unit_map]
+        self.patch_object(
+            handlers.ch_core.hookenv, 'iter_units_for_relation_name')
+        fake_relation_name = mock.MagicMock()
+        fake_relation_name.unit = 'lb/0'
+        self.iter_units_for_relation_name.return_value = [fake_relation_name]
+        self.patch_object(handlers.api_crud, 'delete_hm_port')handlers.update_controller_ip_port_list()self.leader_set.assert_called_once_with({'controller-ip-port-list': json.dumps([
-                    '2001:db8:42::42',
-                    '2001:db8:42::51',
+                    '2001:db8:42::1',])})def test_render(self):
--
2.25.1

Zaza functional test

functest-run-suite读tests.yaml运行测试，测试的每个象位可独立隔离运行。
https://zaza.readthedocs.io/en/latest/runningcharmtests.html

# octavia测试必须enable SG
juju bootstrap stsstack --no-gui --bootstrap-series focal --config use-default-secgroup=true --config network=zhhuabj_port_sec_enabled --config image-stream=daily --config image-metadata-url=http://10.230.19.58/swift/v1/simplestreams/data/ zhhuabj-sg
juju model-defaults use-default-secgroup=true network=zhhuabj_port_sec_enabled image-stream=daily image-metadata-url=http://10.230.19.58/swift/v1/simplestreams/data/
juju switch zhhuabj-sg# 必须运行在有juju环境的节点上如bastion,
# git clone https://github.com/openstack-charmers/zaza.git
cd ~/charms
# 不能用reactive源码，必须得使用完整的octavia charm
# 因为upload charm /home/ubuntu/charms/octavia for series focal
#git clone https://github.com/openstack/charm-octavia.git charm-octavia
charm pull octavia
cd octavia
tox -e func-noop
source .tox/func-noop/bin/activate
ls tests/bundles/focal-ussuri-ha.yaml
ls tests/bundles/overlays/focal-ussuri-ha.yaml.j2
ls tests/tests.yaml
export OS_VIP00=10.0.0.122         #because we are using zhhuabj-sg
functest-run-suite -b focal-ussuri-ha＃下面是分步运行的步骤，但似乎不work
functest-prepare -m testmodel
juju switch testmodel
export OS_VIP00=10.0.0.122
#注意：也需要调整bundle中的mem大小，以免quota exceed
functest-deploy -m testmodel -b ./tests/bundles/focal-ussuri-ha.yaml
source ~/xxx/openstack/novarc
functest-configure -m testmodel
#run test
functest-test -m testmodel
functest-destroy -m testmodel或
juju bootstrap stsstack --no-gui --bootstrap-series focal --config use-default-secgroup=true --config network=zhhuabj_port_sec_enabled --config image-stream=daily --config image-metadata-url=http://10.230.19.58/swift/v1/simplestreams/data/ zhhuabj-sg
juju model-defaults use-default-secgroup=true network=zhhuabj_port_sec_enabled image-stream=daily image-metadata-url=http://10.230.19.58/swift/v1/simplestreams/data/

20220106

tox -e func-target ceph:focal-ussuri

20220520 - 调试pip依赖问题

实际上是这个lp bug : https://bugs.launchpad.net/aodh/+bug/1973116
aodh只有在focal yoga中才遇到这个运行’tox -epy39’ hang在那的问题．

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.9 python3.9-dev python3.9-distutils -y
alias python=python3.9
alias python3=python3.9
sudo apt install build-essential -y
sudo apt install python3-pip python3-dev python3-nose python3-mock -y
sudo apt install python3-setuptools libpython3-dev libssl-dev \libxml2-dev libxslt-dev libxslt1-dev libpq-dev git git-review \libffi-dev gettext graphviz libjpeg-dev zlib1g-dev psycopg2-binary -y
sudo apt install postgresql-server-dev-all libmysqlclient-dev -y
git clone https://opendev.org/openstack/aodh.git -b stable/yoga
cd aodh
tox -e py39#sudo dpkg -i --force-overwrite /var/cache/apt/archives/python3-distutils_3.10.4-0ubuntu1_all.deb
#sudo apt-get -f install

那是因为https://opendev.org/openstack/requirements/raw/branch/stable/yoga/upper-constraints.txt　中缺少了：

setuptools===60.9.3;python_version=='3.9'

所以focal master与focal yoga都使用了setuptools=62.1.0, 而这个62.1.0和testr一起工作没问题，但和non-testr一起工作则有问题．刚好master分支才开始使用testr (https://opendev.org/openstack/aodh/commit/63c3466f291996c3d5f99e307f8784abeeb1ef78).
所以这个focal master没这个问题，而focal yoga则有这个问题．
鉴于修改openstack requirement工程估计不可行，所以提了一个patch来修改aodh自己的requirement.txt (https://review.opendev.org/c/openstack/aodh/+/842697)

tox -e generate
.tox/generate/bin/generate-constraints -b blacklist.txt -r global-requirements.txt -p python3.6 -p python3.7 -p python3.8 -p python3.9 '>' upper-constraints.txt

其他测试手段：

wget https://releases.openstack.org/constraints/upper/yoga -O yoga.txt
.tox/py39/bin/pip install -r yoga.txt
.tox/py39/bin/pip install -r requirements.txt
.tox/py39/bin/pip install -r test-requirements.txt
sudo apt install subunit -y
.tox/py39/bin/python -m subunit.run discover -t ./ ./aodh/tests/unit |subunit2pyunit# run-tests.sh
sudo apt install python3-os-testr -y
export OS_TEST_PATH=aodh/tests/unit
.tox/py39/bin/python setup.py testr --slowest --testr-args="--subunit $*" | subunit-trace -f

也试过二分，这个问题与二分无关：

二分法：https://zhhuabj.blog.csdn.net/article/details/53676439
git log --tags --simplify-by-decoration --pretty="format:%ci %d"
git log --since=2015-01-27 --before=2015-05-04 --oneline src/qemu/qemu_migration.c |wc -l# 13.0.0 is xena, 14.0.0 is yoga
git bisect start
git bisect bad 14.0.0
$ git bisect good 13.0.0
Bisecting: 5 revisions left to test after this (roughly 3 steps)
[9fa5ad045b41f90a1f80aca5d8a46a5abbd848c9] Merge "Introduce Guru Meditation Reports into Aodh"$ git bisect bad
Bisecting: 2 revisions left to test after this (roughly 1 step)
[0d6c43811dc1b531c38bf1fd9c4d345f30ac16b5] Add Python3 yoga unit tests