解决octavia failed to run

octavia failed to run

前言

如果没有特别标注，所有的命令行都是使用admin账户执行的。

问题描述

通过如下调用octavia的api创建loadbalance

@token = gAAAAABe6y8up_ZdUS43Lfe8AFERncGtZFK6IHrnd-MSgbE5U-xKg-6N04eUV3tNwlKEAiZVNIl6I0NVvjMT97CygiJL-7XFciQHl0vWdd2rTQWtaDR9w04pJtk1DYBi7vjcq0VivngYMrn2MhOJmf3X3Fss9kV2BY9m19v-A6mba9CYVPE_NHY
@endpoint = http://10.8.8.11:9876#### create loadbalancer
POST {{endpoint}}/v2.0/lbaas/loadbalancers
X-Auth-Token: {{token}}
Content-Type: application/json{"loadbalancer": {"description": "loadbalancer_created_by_gxj_for_test_purpose",# "flavor_id": "30ad863a-25dc-425a-8ebe-e29f361cb777","admin_state_up": true,"vip_subnet_id": "3aa77344-3086-4d2c-aa1a-27c997c92840","provider": "octavia","name": "loadbalancer_created_by_gxj_for_test_purpose"}
}

调用后成功返回

HTTP/1.0 201 Created
Content-Length: 681
Connection: keep-alive
Content-Type: application/json
Date: Thu, 18 Jun 2020 09:34:14 GMT
keep-alive: timeout=4
Proxy-Connection: keep-alive
Server: WSGIServer/0.1 Python/2.7.12
X-Openstack-Request-Id: req-db7143a3-d2aa-450e-8414-1b2fa257e974{"loadbalancer": {"provider": "octavia","flavor_id": "","description": "loadbalancer_created_by_gxj_for_test_purpose","provisioning_status": "PENDING_CREATE","created_at": "2020-06-18T09:34:14","admin_state_up": true,"updated_at": null,"vip_qos_policy_id": null,"vip_subnet_id": "3aa77344-3086-4d2c-aa1a-27c997c92840","listeners": [],"vip_port_id": "34f53806-1c12-48a5-8724-d965cb74cf7e","vip_network_id": "0c1c8b1d-6515-4e5c-bba2-4a5e3ed80084","vip_address": "10.0.0.29","pools": [],"project_id": "55982e2c523149158a6cc839ce2eece2","id": "59b14489-3f4f-49dc-8533-29fc50377eb9","operating_status": "OFFLINE","name": "loadbalancer_created_by_gxj_for_test_purpose"}
}

但是过一段时间去检查loadbalancer的状态的时候发现是error。

问题排查

大致流程

octavia大致的workflow(工作流)
查看octavia-api的日志
查看octavia-controller-worker的日志

octavia大致的workflow

图是官方的

相信参考OpenStack Docs: Octavia v0.5 Component Design

查看octavia-api的日志

通过vi /var/log/octavia/api.log查询loadbalancer的id发现下面的日志

2020-06-18 17:34:14.880 24792 INFO octavia.api.v2.controllers.load_balancer [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Sending created Load Balancer 59b14489-3f4f-49dc-8533-29fc50377eb9 to the handler

通过这边发现来自client的创建请求已经被send出去给controller-worker了，也就是说octavia这边没问题,但是往上面看了点，发现了一些error

2020-06-18 17:34:03.665 24792 DEBUG octavia.common.keystone [req-72f1be5c-a7e3-42bc-9590-032ec02a1b0a - 55982e2c523149158a6cc839ce2eece2 - default default] Request path is / and it does not require keystone authentication process_request /usr/lib/python2.7/dist-packages/octavia/common/keystone.py:67
2020-06-18 17:34:13.943 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension security-group found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:13.950 24792 DEBUG neutronclient.v2_0.client [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Error message: {"message": "The resource could not be found.<br /><br />\nExtension with alias dns-integration does not exist\n\n", "code": "404 Not Found", "title": "Not Found"} _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
2020-06-18 17:34:13.951 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension dns-integration is not enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:70
2020-06-18 17:34:13.957 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:13.965 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension allowed-address-pairs found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.018 24792 DEBUG octavia.db.repositories [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Checking quota for project: 55982e2c523149158a6cc839ce2eece2 object: <class 'octavia.common.data_models.LoadBalancer'> check_quota_met /usr/lib/python2.7/dist-packages/octavia/db/repositories.py:313
2020-06-18 17:34:14.047 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension security-group found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.052 24792 DEBUG neutronclient.v2_0.client [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Error message: {"message": "The resource could not be found.<br /><br />\nExtension with alias dns-integration does not exist\n\n", "code": "404 Not Found", "title": "Not Found"} _handle_fault_response /usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:259
2020-06-18 17:34:14.052 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension dns-integration is not enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:70
2020-06-18 17:34:14.057 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension qos found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.063 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension allowed-address-pairs found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66
2020-06-18 17:34:14.070 24792 DEBUG octavia.network.drivers.neutron.base [req-db7143a3-d2aa-450e-8414-1b2fa257e974 - 55982e2c523149158a6cc839ce2eece2 - default default] Neutron extension project-id found enabled _check_extension_enabled /usr/lib/python2.7/dist-packages/octavia/network/drivers/neutron/base.py:66

这个error应该是请求的这个资源55982e2c523149158a6cc839ce2eece2不存在。我感觉这是个网络相关的资源于是走了下面的一个命令

root@ctl01:~# openstack network list | grep 55982e2c523149158a6cc839ce2eece2
| 26aef702-6d8e-4384-9c87-5d7c851021ac | HA network tenant 55982e2c523149158a6cc839ce2eece2 | 022fb016-e250-42ad-95e2-8ba4f26350d5

实际上这串uuid是某个租户，想当然往往会出乎意外
于是乎，我尝试了下


root@ctl01:~# openstack project list | grep 55982e2c523149158a6cc839ce2eece2
| 55982e2c523149158a6cc839ce2eece2 | admin                                                            |
| 9d99f566b76246d4b72722b9882c245f | 55982e2c523149158a6cc839ce2eece2-b77e26e9-a4a6-4773-a42d-7ef029d |

发现这个貌似应该是admin的project_id. 然而被没有软用，细看日志发现自己漏了最关键的东西nExtension with alias dns-integration does not exist, google下这个应该是neutron dns as a service

查看octavia-controller-worker的日志

用vim 打开controller-worker的日志文件"/var/log/octavia/worker.log"根据上一次创建失败的loadbalancer的id查找有用日志，下面是相关的日志。

1223 2020-06-18 17:34:14.922 3374787 INFO octavia.controller.queue.endpoint [-] Creating load balancer '59b14489-3f4f-49dc-8533-29fc50377eb9'...
1224 2020-06-18 17:34:14.984 3374787 INFO octavia.controller.worker.tasks.database_tasks [-] Created Amphora in DB with id ee934942-fc30-44ef-b834-e2dda0cd650f
1225 2020-06-18 17:34:15.031 3374787 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
1226 2020-06-18 17:34:15.032 3374787 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
1227 2020-06-18 17:34:15.032 3374787 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
1228 2020-06-18 17:34:15.032 3374787 INFO octavia.certificates.generator.local [-] No Passphrase found for CA Private Key, not using one.
1229 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver [-] Nova failed to build the instance due to: Network 8a588b4c-06ae-4ba2-ae74-2a     8a6ed687ef could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0): BadRequest: Network 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef      could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1230 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver Traceback (most recent call last):
1231 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver   File "/usr/lib/python2.7/dist-packages/octavia/compute/drivers/nova_driver.py"     , line 154, in build
1232 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver     availability_zone=CONF.nova.availability_zone
1233 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 1313, i     n create
1234 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver     return self._boot(response_key, *boot_args, **boot_kwargs)
1235 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver   File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 772, in      _boot
1236 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver     return_raw=return_raw, **kwargs)
1237 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 366, in _crea     te
1238 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver     resp, body = self.api.client.post(url, body=body)
1239 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 310, in      post
1240 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver     return self.request(url, 'POST', **kwargs)
1241 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 83, in requ     est
1242 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver     raise exceptions.from_response(resp, body, url, method)
1243 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver BadRequest: Network 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HT     TP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1244 2020-06-18 17:34:15.561 3374787 ERROR octavia.compute.drivers.nova_driver
1245 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks [-] Compute create for amphora id: ee934942-fc30-44ef-b834-e2dda0cd650     f failed: ComputeBuildException: Failed to build compute instance due to: Network 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HTTP 400) (Req     uest-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1246 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks Traceback (most recent call last):
1247 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks   File "/usr/lib/python2.7/dist-packages/octavia/controller/worker/tas     ks/compute_tasks.py", line 99, in execute
1248 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks     server_group_id=server_group_id)
1249 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks   File "/usr/lib/python2.7/dist-packages/octavia/compute/drivers/nova_     driver.py", line 160, in build
1250 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks     raise exceptions.ComputeBuildException(fault=e)
1251 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks ComputeBuildException: Failed to build compute instance due to: Networ     k 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1250 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks     raise exceptions.ComputeBuildException(fault=e)
1251 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks ComputeBuildException: Failed to build compute instance due to: Networ     k 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef could not be found. (HTTP 400) (Request-ID: req-d8b979fe-0cc5-4b3a-a1aa-3d7e457ca8c0)
1252 2020-06-18 17:34:15.561 3374787 ERROR octavia.controller.worker.tasks.compute_tasks
1253 2020-06-18 17:34:15.563 3374787 WARNING octavia.controller.worker.controller_worker [-] Task 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-cert-co     mpute-create' (2f7f05b0-c7a3-4fa1-94f2-584c70ca20d1) transitioned into state 'FAILURE' from state 'RUNNING'
1254 8 predecessors (most recent first):
1255   Atom 'STANDALONE-octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'server_pem     ': 'gAAAAABe6zUXLP9ofIXoOlDm1_OLc5arzZh_Bo2aunp_iGB9qiRS8X-O79tIYMY24-zhg-MD8JBpWQsZ9hEQsEFYLKaVKjSKUNX9oRkjeyWLZXeAyKhYJ6GN6tndes2LMSL0GrfHqs27fg2UlYuQ0F     cLMpfEKoLXLLc7rqIsHdk1bMTIc0uMAAPE4b4QPPH3G0ta5Mk5uBxrrf5lgNAusjdy4Xygvzu5kXlGXrov50sVdQIINGIbq9GwL6hJYG2SHHz0w2-m0EAG6i_31zDYL76D5gVcCVSt0-pKFOdydOe1am23     Gels0RunQgXhNnQ6yiZJf0L1T9XsrkyFP9u6QmCe5cGF48u0Ddgaa9qOXNuOtLLlWFpkmMQbHOOpDMt96w-3ra149J4R39HMQc8DJdBhE5m-x6XxbmCzojPrFtbO0cyGn3pnkhClPdgsGZtuw5rUFm5SQu     Yw76HuGB2vxe5qeN6JVccGHhJJUum4XRZHmgJEV7kkxeYz0JYwM2TUDYOdOxten3Jb55-YoaB4X0dcamdx6InFYJHueG8VH7l0BPk5OjxHzyWsOfj5cFiLlng3MdqWa3Lw_zUYACF9HHkUycHJ1v-NNzWf     VCFo-8yaDpb6I1tCtwrr6fBdv0x0O_L0XTRNnHS2uztHneIFcXbAfj0gWz1Y2yhpJtLR69PgOM_Qg6RcAZ1fFv4QBynPJ2Zr1D6wJKgz0yJygHrW4LGYrOdnHPHugIKn38xP68qzBNpg-BCqNsjmepPJvx     cJXh9XVrvFKyYkrr7uD5OGuo1yHAXXK4Ur29H9-O4qMZICllHWs7tXa3gKKLqt2jXHYheeIoOwC49bieaLw4RLVuKaQoxrHcqezYeRCtuUo0azZuqP3zdTjO_ywGniw1U0zL5Un5-siO1ukqiadAQOByTT     gd7hpjEl6CW85sukEjZRSOpDMikg1uodBN4ewTZa_J-u4ujEJaMwEnnjqepwXpfQ0Qbzq3ZtXq-g1_q9GJttYGEtyJqTioq6YS5jSUXX4d039NjtsLU0b_voEbXBon6bwZdCTnUEnHQ7kNqdA_Zm1oQMZX     hbAhRWc_chzXx_75TszeRumBwo5DJbmfxuj4BG7IQgtuY-26gH8TZEVVGDNvfNIKXZD-faWR9UXw5KU3SpKU1QYGM5r1oRypRH0NUka4pk-pJzwBpfWZg3gy1-yFmblyGr441-Pfm2cWSXSP0f1oB2mhNf     WJWwKGCKD3f27q5E5ln4guOwPHgg4Ioy3uY8QwfQz_LKlY3UmGr-BMsfi2cfxYru76fAKxUf1V5nL4Bn4Lg2a_pa2y2d2c6v2JBWZZnmwR-yXO1Mxq4e8iock-2FNVlHZuRi2xwNPWn5TqlDaqoxbYmPpD     Deonecix7Q2po_TWjFnGHAMZzX0qka26Sm4yd6JKtIdfakhqakTma_EqmxM2L9HkldLbljCkn9Q8tLsiv4G4x1lz8ZSETjNCflZmuGjLUbXBVCEQl2I398j-XOZ4F1MBSc6hppg9_MwzF17i8w29T3PxUd     jNjA47e1DCb-tD8NnF43BuLufkOzzeRBs5CBXso9SzOTJMtA74EeVzMsDRaFyjZGdWerfKqNisfpzreM1pgLRaM_4BcqPOg5OcaGHrWHhGUV0ov5dzfkzPqskHS-KeI2nT5G3jl37QGfNleN6oBkOArkJz     RJ60Q8xlnqjGex89wClAV_dWKVs7XoVxdOvsOo7eskXCsFxNADcynAGBDxK4CKfnZkcN5cdCwyl15tiLp487XKW-axXo2Oe49QlN6p5DLK-OgBdh4bZdt6jdbx7PvzoXNx2HtELjPkDvplbYz7_bwcZmHk     Tbd7h1iktKMSv5f2I4zTD7CtJFGMVaN1kmTclEgh1KkQUVF91VxzB6aJ24ZFqoUsP9F_dB3cSk2kQ1TpvJlhfj00aVuLnOnD6cgaiIWQ_Dn11rqodvRdYGS7lxTmJhiwRSRkQjnIIVHT7En3k1Hymx3lIm     1EgpiQ1lVveaebj440UdN7dPAlQ6iG5XcRE3ZTGzdxAuV7adJZsI-9NNO6mOzv4bcOsYchDxhEgn_eQW0MuKhafe3-yVt8TrjvW0w4r0J0QaEC0ZxrLiIBFtnen7LVkAoQNGv2u5t-tb6fGGCDD9gFNtwK     dLMC5TR-EmtGbWJehS4bVPR7URTi7czoI0S3GFSXVsqKFQRXRaVqw9uF8OZZFwRYQYKJQg2YnwqYhywfZdl2mMrLxyGeAOe5WqOzI_KeE1fros_l95FinwTbz-i-mm0mO-VDGKhnFcgk2KYQl-w71ywMgp     NKXhgPmLbDFpzButkHFhF618oOc__L4ZvgFczu2YIv-_ZNc2y7NA9939cGUV45hFfzbA8JkmZIgn-80dSVhGe0cmGkeHqsETWFuryHQtA1TWnWng8VrR5GPVSFXvwj4P6IJJsBC9uPHPi8M3pRcN1gdiAq     Nzq1Q4I6ht3aNpPEz16n2dJ56tMTxY30PoH_swQI4KyN4LRsrBmFUA9e4vnkA4V05xZxCdhTbmmbDvJt3Ncej06935biSR_oPgOl6A-IoW7-jZvoLxD0dkGkYQ-NrBhiO0Qxx0iCmtVME4xiSH26N3RS_K     8_odfZSSAOme1-UDqJ0soaysnxnZiE2ioA3hgku-Ns9K29ZTqBDDb68GfTg4jPzz3jeNUI8GRSuiKbqAf1et6swZHYwsfg559XIxLnWDdyRacXwic2-mA5QSGzT8uEeYEQfsxYplOGAIO4isLBiCW6hwXW     ZcgC8gUpCgbdpUHaxGCxBLrYl8HY62wORzcTkOf95oEQ3oJO9GgWX_gx6LQZh-05FvUnWO_IxGd8-hzvExIZ_DqL3I139Y9vy2590oxRsI4AmdOpuS47_jD0KgDGjj_4qF2bqhvrz2HuYL1eGFVGEyT1jD     RwgkvfO6qpWBfs-9-utmKAgUg0dvXj-VrZDlHNUBLuhpk8yJNUM8NHVLjpPrJpbCGbZsLw-9r0nG3U4hf8zJSnk_0npEiTsuuYK0Y_Fz1IRyzwGoYpIc7ORuBzYK3nG_AAxCIGmrI2PMO7FeKIgWaanIB0     VRnRkRf9gRv4XkzQnGPYPYFrt6zDxizklpElVK6oasZqOaVT-Oxpw6Uu5v6UR_EFzTDiycDmRqwJfxIRw_izT3BD4vP4LGtvoexq-V58z4Ob3KRUYSOYh-bO3wueZE7dd5U8R98hWqLPjgc6E76vQrpudk     sUa1j_5hRt4jMJPWf3eOPHI54Npdj9Jo1tyfGaNDxw_zOIZRxCpfFunL7w4AGsgXPsRKQIhsuLvKI01W2IX5sJ-dJIbQCfRYxQn_nIM_RLxOCY1lj-aL1Qc3Yn_gECx-dUBqyKoPnTiBVF20qlpOgq-6dA     -OUa31nAK240KEAIaIXerai-ucltQsnJMFEwgBJduxJPT7d2IIszqaoBJyFDavYKbgBaA2EWokZCv5ur7-UkXRvdkf9MuvjczwUKdLXIXrlyHW8QXtnJN7BtQSoU3MulPQZkcaGXdUOBHHPi-uB6TM5zzi     Gypk1N7F3BvMU0yjkraKYesJy4sCbhFk0WwJOapBVQrmQvsyzcCDpdWUvGqp581Sm9YOJ2toeYRCIT63CaJqdlEEOWBXB3MLgoAJDPkMMorH1VkMPWUhGZRMxy_p9bTVMoj6Kd3-z8d70NjGRRPRmxeDli     Dq5oDwSdcslGC0yepxNRgRmel5VNmugWxpbWFjrr-cURMQ0BLlWhj7YUoh_7oPw0-_8NHuSaVxtk85p7bBpXXo-9qGrNKt8nHNoPtTj0uV2q4w9ROAYFNU8iHcsjVItfV9mG3Bl4rrXUZ-MPod3u8XuTVV     Fty5f8Qic3JS66LWi8vKt2SbRlto9Ai3iYSpTJa81k_IHHVQWdJ7L7waakSuiGjjKOL37f7kQRNLgtwb5lWYXjLPih39PSdHiRVT_huboQnXfhVrNMAH3eq76BBNhgxGyuZaxSqhmONee7_1LnVA5HZMx7     aAJOBkloG56jZNOvnTqbiDlgGC4neJSm-tiSIiHo4u6KONoSRLH76OL2EiVpTGRhQN3wRQkPqzvDOvUtpwJ46_SbH-cbLc-GPFuz4WlGL6dAtQ7Z9KW9TosOC79l-skMeq2EJyTPgedVoedsA01cPiXTz9     q4xYG3Hi2iBc1No-809jYSg4hzrkTUl_kv4nwOsQwU8ykQxIXr3EYgUiJ7AxLXU4ZYf3lRTXG7drE3a0n_itWG-7oOMKnfxoExo-TjC8cUJ0KUHq-ydZ3VskhhlFkSLiH-Rxs4DxzrNlRQN1q-i1O98eYN     kJiOOuP4YJRSF6PF9fJX8-lQIsopvrlsx2-x4Rs_HeObEgQiW0s2RULRE16d8XaJC4ko1Rodx7oW3ZS0olIzqYT1qqhqa4oqoQjGWxkoWmRkPG6nZK4MAZkebGLNNtwyuss77HQ7UrBCidaazKhQEKFPVK     dphdNY4SA=', 'amphora_id': u'ee934942-fc30-44ef-b834-e2dda0cd650f'}, 'provides': None}

根据日志的提示，octavia由于没有找到network id 为8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef的网络导致无法调用nova成功创建amphora虚拟机。
根据下面的命令

root@ctl01:~# openstack network list | grep 3aa77344-3086-4d2c-aa1a-27c997c92840
| 0c1c8b1d-6515-4e5c-bba2-4a5e3ed80084 | tent_net                                           | 3aa77344-3086-4d2c-aa1a-27c997c92840                                                                             |

我们可以看到本套openstack确实没有这个网络，此外我们传入的数据也没有什么跟着个网络id挂钩的。
那么这个网络id，可以的原因便是以下几种

octavia-api处理流程中增加的
octavia-controller-worker中增加的

通过我敏锐的第六感，这两种服务加入这个网络的途径，只有三种：

数据库
第三方服务
配置文件

我们先从配置文件开始，别问我问什么，问就是这个最简单。

配置文件排查

大致流程
- ps -ef | grep target service name碰下运气看能否找到配置文件
- 开启grep大法使用grep target_id -rn config_foler or grep target_id config_file
开始表演
开始ps


root@cmp002:~# ps -ef | grep octavia-worker
octavia  3381004       1  0 May26 ?        00:00:00 octavia-worker: master process [/usr/bin/octavia-worker --config-file /etc/octavia/octavia_manager.conf --config-dir /etc/octavia/conf.d/common --config-dir /etc/octavia/conf.d/octavia-worker --log-file /var/log/octavia/worker.log]
octavia  3381139 3381004  0 May26 ?        01:25:35 octavia-worker: ConsumerService worker(0)
root     4066420 4064201  0 09:38 pts/11   00:00:00 grep --color=auto octavia-worker

找到配置文件/etc/octavia/octavia_manager.conf和配置文件夹/etc/octavia/conf.d/common和/etc/octavia/conf.d/octavia-worker
开启grep大法

root@cmp001:~# grep "8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef" -rn /etc/octavia/
/etc/octavia/octavia_manager.conf:521:amp_boot_network_list = 8a588b4c-06ae-4ba2-ae74-2a8a6ed687ef

运气太好了，第一次尝试就找到了这个网络id
根据这篇文档OpenStack Docs: Load-Balancer-as-a-Service configuration options
配置amp_boot_network_list的作用是(List) List of networks to attach to the Amphorae. All networks defined in the list will be attached to each amphora.

使用ocatavia用户创建一个网络替代这个网络

export OS_IDENTITY_API_VERSION=3
export OS_AUTH_URL=http://10.8.8.11:35357/v3
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=service
export OS_TENANT_NAME=service
export OS_USERNAME=octavia
export OS_PASSWORD=trVvCUyPpHwBNfDppE7ug5we1GRvJyeU
export OS_REGION_NAME=RegionOne
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE="internal"
export OS_CACERT="/etc/ssl/certs/ca-certificates.crt"root@ctl01:~# openstack network create amp_boot_network_1
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | UP                                   |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2020-06-19T03:20:52Z                 |
| description               |                                      |
| dns_domain                | None                                 |
| id                        | 3ca482e2-34a5-4722-a4c5-2d7262e00945 |
| ipv4_address_scope        | None                                 |
| ipv6_address_scope        | None                                 |
| is_default                | False                                |
| is_vlan_transparent       | None                                 |
| mtu                       | 1450                                 |
| name                      | amp_boot_network_1                   |
| port_security_enabled     | True                                 |
| project_id                | cb20ced7c1ea4f43a35eb15f0ec23650     |
| provider:network_type     | vxlan                                |
| provider:physical_network | None                                 |
| provider:segmentation_id  | 45                                   |
| qos_policy_id             | None                                 |
| revision_number           | 2                                    |
| router:external           | Internal                             |
| segments                  | None                                 |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| updated_at                | 2020-06-19T03:20:52Z                 |
+---------------------------+--------------------------------------+

创建完网络后更新/etc/octavia/octavia/octavia_manager.conf里面的amp_boot_network_list项
然后重启所有octavia-worker节点上的octavia-worker服务。
创建新的loadbalancer验证是否成功

验证

创建一个loadbalancer，并查看状态

POST {{endpoint}}/v2.0/lbaas/loadbalancers
X-Auth-Token: {{token}}
Content-Type: application/json{"loadbalancer": {"description": "loadbalancer_created_by_gxj_for_test_purpose",# "flavor_id": "30ad863a-25dc-425a-8ebe-e29f361cb777","admin_state_up": true,"vip_subnet_id": "3aa77344-3086-4d2c-aa1a-27c997c92840","provider": "octavia","name": "loadbalancer_created_by_gxj_for_test_purpose"}
}root@ctl01:~# openstack loadbalancer list | grep load
| 9bae06de-d0b9-4cce-9198-389a18de9fcd | loadbalancer_created_by_gxj_for_test_purpose | 55982e2c523149158a6cc839ce2eece2 | 10.0.0.8    | ACTIVE              | octavia  |

创建成功，问题解决