虚机获取不到IP报错No lease, failing (by quqi99)

作者：张华发表于：2013-12-22
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

( http://blog.csdn.net/quqi99 )

虚机获取不到IP，排除32位系统使用64位镜像之后，查看其console log的信息如下：

Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending discover...
Sending discover...
No lease, failing
WARN: /etc/rc3.d/S40-network failed

解决办法：

iptables -A POSTROUTING -t mangle -p udp --dport 68 -j CHECKSUM --checksum-fill

https://bugs.launchpad.net/neutron/+bug/1722584

https://lore.kernel.org/patchwork/patch/824819/

--checksum-fill

Compute and fill in the checksum in a packet that lacks a checksum. This is particularly useful, if you need to work around old applications such as dhcpclients, that do not work well with checksum offloads, but don't want to disable checksum offload in your device.

2014-01-25添加：

虚机没有IP还有一种情况，在ovs中dhcp的port的vlan号变成了4095，这个是不对的。

2014-08-28添加：

今天帮一个同事线上远程调试一个环境，它用的gre, 忽然说虚机取不到ip，查来查去，原因是它中途把一个计算节点的ip改了，然后把local_ip参数也给变了，造成那台计算节点自己和自己建立了遂道。这种人为的问题真难查。

2014-08-29, 这位同事遇到的可能是这样一个问题 (https://lists.launchpad.net/openstack/msg23595.html, https://bugs.launchpad.net/neutron/+bug/1179223 )

2014-10-17, metadata服务不通也会造成此问题

所以解决此问题的关键是看console log, 具体分析

$ sudo dhcpclient eth0

$ sudo tcpdump -ni eth0 port 67 or port 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:06:00.684853 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 28:d2:44:52:31:1d, length 300
11:06:00.686075 IP 192.168.99.1.67 > 192.168.99.135.68: BOOTP/DHCP, Reply, length 300

1, sudo ovs-ofctl dump-flows br-int

2, sudo ovs-appctl bridge/dump-flows br-int # can show some HIDDEN rules as well

实例：

hua@node1:/bak/openstack/devstack$ nova reboot testAttach
Request to reboot server <Server: testAttach> has been accepted.

qvo口有dhcp请求。
hua@node1:/bak/openstack/devstack$ sudo tcpdump -ni qvo3c64d4e8-ad port 67 or port 68
tcpdump: WARNING: qvo3c64d4e8-ad: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qvo3c64d4e8-ad, link-type EN10MB (Ethernet), capture size 65535 bytes

14:39:59.180107 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:5b:08:a3, length 294

但dhcp port没有收到dhcp请求，那肯定是br-int里的流规则出问题了。
hua@node1:/bak/openstack/devstack$ sudo ip netns exec qdhcp-15101061-5784-422b-8c6b-7885b3616d07 tcpdump -ni tap36af97e6-50 port 67 or port 68
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap36af97e6-50, link-type EN10MB (Ethernet), capture size 65535 bytes

hua@node1:~$ sudo ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0x80d224499fbdf5f8, duration=1890.568s, table=0, n_packets=0, n_bytes=0, idle_age=1890, priority=10,icmp6,in_port=10,icmp_type=136 actions=resubmit(,24)
cookie=0x80d224499fbdf5f8, duration=1890.795s, table=0, n_packets=0, n_bytes=0, idle_age=1890, priority=3,in_port=7,dl_vlan=1003 actions=mod_vlan_vid:2,NORMAL
cookie=0x80d224499fbdf5f8, duration=1890.563s, table=0, n_packets=0, n_bytes=0, idle_age=1890, priority=10,arp,in_port=10 actions=resubmit(,24)
cookie=0x80d224499fbdf5f8, duration=2229.258s, table=0, n_packets=0, n_bytes=0, idle_age=2229, priority=2,in_port=7 actions=drop
cookie=0x80d224499fbdf5f8, duration=2229.305s, table=0, n_packets=422, n_bytes=71584, idle_age=39, priority=0 actions=NORMAL
cookie=0x80d224499fbdf5f8, duration=2229.303s, table=23, n_packets=0, n_bytes=0, idle_age=2229, priority=0 actions=drop
cookie=0x80d224499fbdf5f8, duration=1890.57s, table=24, n_packets=0, n_bytes=0, idle_age=1890, priority=2,icmp6,in_port=10,icmp_type=136,nd_target=fe80::f816:3eff:fe5b:8a3 actions=NORMAL
cookie=0x80d224499fbdf5f8, duration=1890.566s, table=24, n_packets=0, n_bytes=0, idle_age=1890, priority=2,arp,in_port=10,arp_spa=10.0.1.24 actions=NORMAL
cookie=0x80d224499fbdf5f8, duration=2229.301s, table=24, n_packets=0, n_bytes=0, idle_age=2229, priority=0 actions=drop

hua@node1:~$ sudo ovs-appctl bridge/dump-flows br-int
duration=1918s, priority=10, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=10,icmp_type=136,actions=resubmit(,24)
#qvo的local_vlan=2，所以这是从int-br-phy(in_port=7)进虚机经过br-int时将vlan改成local_vlan
duration=1918s, priority=3, n_packets=0, n_bytes=0, priority=3,in_port=7,dl_vlan=1003,actions=mod_vlan_vid:2,NORMAL
duration=1918s, priority=10, n_packets=0, n_bytes=0, priority=10,arp,in_port=10,actions=resubmit(,24)
duration=2256s, priority=2, n_packets=0, n_bytes=0, priority=2,in_port=7,actions=drop
duration=2257s, priority=0, n_packets=423, n_bytes=71920, priority=0,actions=NORMAL
table_id=23, duration=2257s, priority=0, n_packets=0, n_bytes=0, priority=0,actions=drop
table_id=24, duration=1918s, priority=2, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=10,icmp_type=136,nd_target=fe80::f816:3eff:fe5b:8a3,actions=NORMAL
#允许从qvo口（in_port=10）进br-int也就是出虚机的ARP流量
table_id=24, duration=1918s, priority=2, n_packets=0, n_bytes=0, priority=2,arp,in_port=10,arp_spa=10.0.1.24,actions=NORMAL
table_id=24, duration=2257s, priority=0, n_packets=0, n_bytes=0, priority=0,actions=drop
table_id=254, duration=4478s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x3,actions=drop
table_id=254, duration=4478s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x1,actions=controller(reason=no_match)
table_id=254, duration=4478s, priority=0, n_packets=0, n_bytes=0, priority=0,reg0=0x2,actions=drop

hua@node1:~$ sudo ovs-vsctl show
d186e8ba-b5ce-4062-a50e-202db91f80c1
Bridge br-int
fail_mode: secure
Port "tap36af97e6-50"
Interface "tap36af97e6-50"
type: internal
Port int-br-phy
Interface int-br-phy
type: patch
options: {peer=phy-br-phy}
Port "qr-3233dbfb-2c"
Interface "qr-3233dbfb-2c"
type: internal
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qvo3c64d4e8-ad"
tag: 2
Interface "qvo3c64d4e8-ad"
Bridge br-phy
Port phy-br-phy
Interface phy-br-phy
type: patch
options: {peer=int-br-phy}
Port br-phy
Interface br-phy
type: internal
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "qg-e837ef4b-73"
Interface "qg-e837ef4b-73"
type: internal
Bridge br-tun
fail_mode: secure
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
ovs_version: "2.0.2"

20190507更新:

今天遇到一个虚机无法获得IP的问题, 使用tcpdump命令(tcpdump -enli bond0 udp port 4789)在计算节点的bond0上能看到ping后的ICMP响应, 但无法看到DHCP request 与DHCP reply,

05:27:24.003173 fa:16:3e:a0:5a:17 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:a0:5a:17, length 300
05:27:26.750370 fa:16:3e:38:6a:e8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.222.100.1 tell 10.222.100.10, length 28

在bond0.60, qbr, qbo, tap等的网卡上能看到DHCP request不能看到DHCP reply. 在dhcp-agent节点上的tap上也无法看到request与reply. 在一个好的节点上却能看到下列的tcpdump输出:

05:46:47.092678 fa:16:3e:6a:14:36 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:6a:14:36, length 300
05:46:47.092886 fa:16:3e:5a:d1:0b > fa:16:3e:6a:14:36, ethertype IPv4 (0x0800), length 390: 10.222.100.11.67 > 10.222.100.24.68: BOOTP/DHCP, Reply, length 348

客户使用的是mallonax的网卡, 默认打开了offload. 换成不支持offload的网卡能解决问题, 使用hwe kernel (sudo apt-get install --install-recommends linux-generic-hwe-16.04)也能解决问题(git log --no-merges --oneline origin/hwe...origin/master drivers/net/ethernet/mellanox/mlx5/ |grep offload).

参考：http://techbackground.blogspot.kr/2013/05/debugging-quantum-dhcp-and-open-vswitch.html

http://yeasy.blogspot.kr/2013/06/openstack-vm-cannot-get-ip-by-dhcp-with_4553.html