RYU控制器的学习笔记（一） ryu.app.rest

参考链接：https://www.cnblogs.com/goldsunshine/p/11720310.html

1.默认的流表

当你启动任何一个ryu app之后，交换机的流表就被设置为默认的下面的内容

root@user-NF8480M5:/home/user/qyq/dragonflow/main# ovs-ofctl dump-flows ckltest-br
NXST_FLOW reply (xid=0x4):cookie=0x0, duration=1.737s, table=0, n_packets=0, n_bytes=0, idle_age=1, priority=1,arp actions=CONTROLLER:65535cookie=0x0, duration=1.737s, table=0, n_packets=0, n_bytes=0, idle_age=1, priority=1,ip actions=dropcookie=0x0, duration=1.737s, table=0, n_packets=0, n_bytes=0, idle_age=1, priority=0 actions=NORMAL

2.默认流表下的流量转发

以同一个ovs上的主机Aping主机B为例，默认的话是ping不通的，仅仅增加arp表项的处理包个数

例如，下面的是192.168.1.2 ping 192.168.0.3,会首先寻找默认网关的mac地址

root@user-NF8480M5:/home/user# tcpdump -i tapq
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapq, link-type EN10MB (Ethernet), capture size 262144 bytes
16:29:36.922760 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:37.939530 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:38.963532 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:39.987755 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:41.011541 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28
16:29:42.035539 ARP, Request who-has 192.168.1.1 tell 192.168.1.2, length 28

刚好被流表接受到6个流量包

NXST_FLOW reply (xid=0x4):cookie=0x0, duration=574.422s, table=0, n_packets=6, n_bytes=252, idle_age=560, priority=1,arp actions=CONTROLLER:65535cookie=0x0, duration=574.422s, table=0, n_packets=0, n_bytes=0, idle_age=574, priority=1,ip actions=dropcookie=0x0, duration=574.422s, table=0, n_packets=0, n_bytes=0, idle_age=574, priority=0 actions=NORMAL

3.CONTROLLER:65535是什么意思？

是指将流量包转发给控制器的意思

4.192.168.1.2 ping 192.168.0.3的提交给控制器的流量包的分析

第一个包（arp表项中有网关的地址）

packet_in_handler!, header_list is
{
'ethernet':
ethernet(dst='fa:c0:29:b8:c6:77',
ethertype=2048,src='86:3c:7c:4b:f1:7a'),
'icmp': icmp(code=0,csum=31873,
data=echo(data='\xce,\xc6^\x00\x00\x00\x00\x05
\x0f\x0f\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14
\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f!"
#$%&\'()*+,-./01234567',id=5136,seq=1),type=8),
'ipv4':
ipv4(csum=13132,dst='192.168.0.3',flags=2,
header_length=5,identification=34055,offset=0,
option=None,proto=1,src='192.168.1.2',tos=0,
total_length=84,ttl=64,version=4)
}

dst_mac是网关的mac地址，

ipv4的源ip是192.168.1.2，目的ip是192.168.0.3

可以看一下网卡的mac

root@user-NF8480M5:/home/user/qyq/dragonflow/main# ip netn exec 82341 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft forever
1040: vtapq@if1041: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state
UP group default qlen 1000link/ether 86:3c:7c:4b:f1:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 192.168.1.2/24 scope global vtapqvalid_lft forever preferred_lft forever
root@user-NF8480M5:/home/user/qyq/dragonflow/main# ip netn exec 82341 arp -a
? (192.168.1.1) 位于 fa:c0:29:b8:c6:77 [ether] 在 vtapq

下面是第一个包的发送过程，192.168.1.2 发送到的192.168.0.3的icmp包，但是mac地址是网关的mac地址

1, packet_in_handler
dp_id is 38074417935428
Router, msg.data
packet_in_handler!, header_list is {'ethernet': ethernet(dst='fa:c0:29:b8:c6:77',ethertype=2048,
src='86:3c:7c:4b:f1:7a'), 'icmp': icmp(code=0,csum=3622,
data=echo(data='\x8dC\xc6^\x00\x00\x00\x00t\x9b\x06
\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15\x16\x17
\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"
#$%&\'()*+,-./01234567',id=23752,seq=1),type=8),
'ipv4':ipv4(csum=12866,dst='192.168.0.3',
flags=2,header_length=5,identification=34321,
offset=0,option=None,proto=1,
src='192.168.1.2',tos=0,total_length=84,ttl=64,version=4)}
IPV4 in header_list
[RT][INFO] switch_id=000022a0e427f444:
Receive IP packet from [192.168.1.2] to an internal host [192.168.0.3].send_arp_request, src_ip is 192.168.0.1, dst_ip is 192.168.0.3[RT][INFO] switch_id=000022a0e427f444: Send ARP request (flood)

注意看倒数第二行，这个时候192.168.0.1（OVS网桥）发送了ARP报文，这个也可以从抓包分析里面得到结果。。

下面对192.168.0.3的网卡进行抓包

17:25:23.233879 ARP, Request who-has 192.168.0.1 tell 192.168.0.1, length 46
（初始化的时候）
17:25:23.241732 ARP, Request who-has 192.168.1.1 tell 192.168.1.1, length 46
（初始化的时候）
17:25:41.149465 ARP, Request who-has 192.168.0.3 tell 192.168.0.1, length 46
17:25:41.149489 ARP, Reply 192.168.0.3 is-at ca:4b:f7:92:f2:ef (oui Unknown), length 28
（第一次ping的时候，OVS需要知道192.168.0.3的MAC地址）
17:25:41.153425 IP 192.168.1.2 > 192.168.0.3: ICMP echo request, id 28032, seq 1, length 64
17:25:41.153453 IP 192.168.0.3 > 192.168.1.2: ICMP echo reply, id 28032, seq 1, length 64
（走下面的流表中被我标注的规则）
17:25:42.144964 IP 192.168.1.2 > 192.168.0.3: ICMP echo request, id 28032, seq 2, length 64
17:25:42.145003 IP 192.168.0.3 > 192.168.1.2: ICMP echo reply, id 28032, seq 2, length 64

下面是流表中的规则，其中我做记号的几行是被实际流量包走的规则

cookie=0x1, duration=418.888s, table=0, n_packets=0, n_bytes=0,idle_age=418, priority=1037,ip,nw_dst=192.168.0.1 actions=CONTROLLER:65535
cookie=0x2, duration=418.880s, table=0, n_packets=0, n_bytes=0,
idle_age=418, priority=1037,ip,nw_dst=192.168.1.1 actions=CONTROLLER:65535# 192.168.1.2 ping 192.168.0.3是直接走的下面这两行
cookie=0x2, duration=400.960s, table=0, n_packets=12, n_bytes=1176,
idle_timeout=1800, idle_age=2, priority=35,ip,
nw_dst=192.168.1.2 actions=dec_ttl,mod_dl_src:fa:c0:29:b8:c6:77,
mod_dl_dst:86:3c:7c:4b:f1:7a,output:1029cookie=0x1, duration=0.402s, table=0, n_packets=12, n_bytes=1176,
idle_timeout=1800, idle_age=2, priority=35,ip,
nw_dst=192.168.0.3 actions=dec_ttl,mod_dl_src:46:0e:bf:4a:7c:11,
mod_dl_dst:ca:4b:f7:92:f2:ef,output:1028cookie=0x1, duration=418.888s, table=0, n_packets=0, n_bytes=0,
idle_age=418, priority=36,ip,nw_src=192.168.0.0/24,nw_dst=192.168.0.0/24
actions=NORMAL
cookie=0x2, duration=418.880s, table=0, n_packets=0, n_bytes=0,
idle_age=418, priority=36,ip,nw_src=192.168.1.0/24,nw_dst=192.168.1.0/24
actions=NORMAL
cookie=0x1, duration=418.888s, table=0, n_packets=1, n_bytes=98,
idle_age=400, priority=2,ip,nw_dst=192.168.0.0/24 actions=CONTROLLER:65535
cookie=0x2, duration=418.880s, table=0, n_packets=1, n_bytes=98,
idle_age=400, priority=2,ip,nw_dst=192.168.1.0/24 actions=CONTROLLER:65535# arp走的是下面的这行规则
cookie=0x0, duration=495.466s, table=0, n_packets=47, n_bytes=1974,
idle_age=0, priority=1,arp actions=CONTROLLER:65535cookie=0x0, duration=495.466s, table=0, n_packets=9, n_bytes=756,
idle_age=4666, priority=1,ip actions=drop
cookie=0x0, duration=495.466s, table=0, n_packets=12, n_bytes=1062,
idle_age=2666, priority=0 actions=NORMAL

下面是第二个到达OVS的包，是192.168.0.3向192.168.0.1的arp响应报文

packet_in_handler!, header_list is {'arp': arp(dst_ip='192.168.0.1',dst_mac='46:0e:bf:4a:7c:11',
hlen=6,hwtype=1,opcode=2,plen=4,proto=2048,src_ip='192.168.0.3',
src_mac='ca:4b:f7:92:f2:ef'),
'ethernet': ethernet(dst='46:0e:bf:4a:7c:11',ethertype=2054,src='ca:4b:f7:92:f2:ef')
}

这个包是上一次缓存了ICMP包，并向所有网口洪泛了的ARP包，当192.168.0.3接收到ARP请求报文时，会发送ARP响应包给OVS

    def _packetin_to_node(self, msg, header_list):if len(self.packet_buffer) >= MAX_SUSPENDPACKETS:self.logger.info('Packet is dropped, MAX_SUSPENDPACKETS exceeded.',extra=self.sw_id)return# Send ARP request to get node MAC address.in_port = self.ofctl.get_packetin_inport(msg)src_ip = Nonedst_ip = header_list[IPV4].dstsrcip = ip_addr_ntoa(header_list[IPV4].src)dstip = ip_addr_ntoa(dst_ip)address = self.address_data.get_data(ip=dst_ip)if address is not None:log_msg = 'Receive IP packet from [%s] to an internal host [%s].'self.logger.info(log_msg, srcip, dstip, extra=self.sw_id)src_ip = address.default_gw  # 192.168.1.2, 192.168.1.1else:route = self.routing_tbl.get_data(dst_ip=dst_ip)if route is not None:log_msg = 'Receive IP packet from [%s] to [%s].'self.logger.info(log_msg, srcip, dstip, extra=self.sw_id)gw_address = self.address_data.get_data(ip=route.gateway_ip)if gw_address is not None:src_ip = gw_address.default_gwdst_ip = route.gateway_ipif src_ip is not None:self.packet_buffer.add(in_port, header_list, msg.data)self.send_arp_request(src_ip, dst_ip, in_port=in_port) // 这一行是重点self.logger.info('Send ARP request (flood)', extra=self.sw_id)

下面是第二个到达OVS的包，是192.168.0.3向192.168.0.1的arp响应报文

packet_in_handler!, header_list is {'arp': arp(dst_ip='192.168.0.1',dst_mac='46:0e:bf:4a:7c:11',
hlen=6,hwtype=1,opcode=2,plen=4,proto=2048,src_ip='192.168.0.3',
src_mac='ca:4b:f7:92:f2:ef'),
'ethernet': ethernet(dst='46:0e:bf:4a:7c:11',ethertype=2054,src='ca:4b:f7:92:f2:ef')
}

处理程序是自己写的

    def packet_in_handler(self, msg, header_list):# Check invalid TTL (for OpenFlow V1.2/1.3)ofproto = self.dp.ofprotoif ofproto.OFP_VERSION == ofproto_v1_2.OFP_VERSION or \ofproto.OFP_VERSION == ofproto_v1_3.OFP_VERSION:if msg.reason == ofproto.OFPR_INVALID_TTL:self._packetin_invalid_ttl(msg, header_list)return# Analyze event type.if ARP in header_list:self._packetin_arp(msg, header_list)return.............

self._packetin_arp

    def _packetin_arp(self, msg, header_list):src_addr = self.address_data.get_data(ip=header_list[ARP].src_ip)if src_addr is None:return# case: Receive ARP from the gateway#  Update routing table.# case: Receive ARP from an internal host#  Learning host MAC.gw_flg = self._update_routing_tbl(msg, header_list)if gw_flg is False:self._learning_host_mac(msg, header_list)..........

应该是receive ARP from an inernal host

    def _learning_host_mac(self, msg, header_list):# Set flow: routing to internal Host.out_port = self.ofctl.get_packetin_inport(msg)src_mac = header_list[ARP].src_macdst_mac = self.port_data[out_port].macsrc_ip = header_list[ARP].src_ipgateways = self.routing_tbl.get_gateways()if src_ip not in gateways:address = self.address_data.get_data(ip=src_ip)if address is not None:cookie = self._id_to_cookie(REST_ADDRESSID, address.address_id)priority = self._get_priority(PRIORITY_IMPLICIT_ROUTING)self.ofctl.set_routing_flow(cookie, priority,out_port, dl_vlan=self.vlan_id,src_mac=dst_mac, dst_mac=src_mac,nw_dst=src_ip,idle_timeout=IDLE_TIMEOUT,dec_ttl=True)self.logger.info('Set implicit routing flow [cookie=0x%x]',cookie, extra=self.sw_id)

下面是第三个到达OVS的包，是192.168.0.3到192.168.1.2的icmp echo response报文

packet_in_handler!,
header_list is
{'ethernet': ethernet(dst='46:0e:bf:4a:7c:11',ethertype=2048,src='ca:4b:f7:92:f2:ef'), 'icmp':
icmp(code=0,csum=16750,
data=echo(data='\xdbg\xc7^\x00\x00\x00\x00\xb7\x07
\x0f\x00\x00\x00\x00\x00\x10\x11\x12\x13\x14\x15
\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'()*+,
-./01234567',id=38639,seq=1),type=0),
'ipv4':
ipv4(csum=12313,dst='192.168.1.2',
flags=0,
header_length=5,identification=51258,
offset=0,option=None,
proto=1,
src='192.168.0.3',
tos=0,total_length=84,ttl=64,version=4)}

这个包的处理程序也是自己写的

下面第四个到达OVS的包，是192.168.1.2向192.168.1.1的arp响应报文

packet_in_handler!, header_list is
{
'arp':arp(dst_ip='192.168.1.1',dst_mac='fa:c0:29:b8:c6:77',
hlen=6,hwtype=1,opcode=2,plen=4,proto=2048,
src_ip='192.168.1.2',src_mac='86:3c:7c:4b:f1:7a'),
'ethernet':
ethernet(dst='fa:c0:29:b8:c6:77',
ethertype=2054,
src='86:3c:7c:4b:f1:7a')
}

5.新增加的规则分析

cookie=0x3, duration=1.625s, table=0, n_packets=1,
n_bytes=98, idle_timeout=1800, idle_age=6,
priority=35,ip,nw_dst=192.168.0.3 actions=dec_ttl, // ttl减少一个，为0的时候丢弃
mod_dl_src:46:0e:bf:4a:7c:11, // tapq1的veth的mac
mod_dl_dst:ca:4b:f7:92:f2:ef, // vtapq1的veth的mac
output:1028 // tapq1cookie=0x4, duration=1.624s, table=0, n_packets=1,
n_bytes=98, idle_timeout=1800,
idle_age=6,
priority=35,ip,nw_dst=192.168.1.2
actions=dec_ttl,
mod_dl_src:fa:c0:29:b8:c6:77,
mod_dl_dst:86:3c:7c:4b:f1:7a,output:1029

这是为什么呢，由于我们是1.2 ping 0.3，先看1.2的mac地址

1040: vtapq@if1041: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue state UP group default qlen 1000
link/ether 86:3c:7c:4b:f1:7a brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.1.2/24 scope global vtapq
valid_lft forever preferred_lft forever

1.2的mac地址是86:3c:7c:4b:f1:7a

0.3的mac地址

1038: vtapq1@if1039: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ca:4b:f7:92:f2:ef brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.3/24 scope global vtapq1
valid_lft forever preferred_lft forever

看一看他们的veth设备的mac地址

1039: tapq1@if1038: <BROADCAST,MULTICAST,UP,LOWER_UP>mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000link/ether 46:0e:bf:4a:7c:11 brd ff:ff:ff:ff:ff:ff link-netnsid 2inet6 fe80::440e:bfff:fe4a:7c11/64 scope link valid_lft forever preferred_lft forever
1041: tapq@if1040: <BROADCAST,MULTICAST,UP,LOWER_UP>
mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000link/ether fa:c0:29:b8:c6:77 brd ff:ff:ff:ff:ff:ff link-netnsid 3inet6 fe80::f8c0:29ff:feb8:c677/64 scope link valid_lft forever preferred_lft forever

看看端口

# ovs-ofctl show ckltest-br
OFPT_FEATURES_REPLY (xid=0x2): dpid:000022a0e427f444
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp
strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst
mod_nw_tos mod_tp_src mod_tp_dst1028(tapq1): addr:46:0e:bf:4a:7c:11config:     0state:      0current:    10GB-FD COPPERspeed: 10000 Mbps now, 0 Mbps max1029(tapq): addr:fa:c0:29:b8:c6:77config:     0state:      0current:    10GB-FD COPPERspeed: 10000 Mbps now, 0 Mbps maxLOCAL(ckltest-br): addr:22:a0:e4:27:f4:44config:     0state:      0speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

6.192.168.1.2对应的tapq上实际捕获的流量分析
ICMP ECHO REQUEST

ICMP ECHO REPLY

7.192.168.0.3对应的tapq1上面实际捕获的流量的分析

ICMP ECHO REQUEST

ICMP ECHO REPLY

8.TTL分析

192.168.1.2 ping 192.168.0.3 第一次到达tapq的时候，如下

然后经过OVS之后，TTL-1 ,MAC地址被修改并转发给tapq1，如下

这样就实现了转发

8.ping默认网关

192.168.0.3 ping 192.168.0.1

代码如下

        if IPV4 in header_list:rt_ports = self.address_data.get_default_gw() # 这个是网关的ipif header_list[IPV4].dst in rt_ports:# Packet to router's port.if ICMP in header_list:if header_list[ICMP].type == icmp.ICMP_ECHO_REQUEST:self._packetin_icmp_req(msg, header_list)returnelif TCP in header_list or UDP in header_list:self._packetin_tcp_udp(msg, header_list)returnelse:# Packet to internal host or gateway router.self._packetin_to_node(msg, header_list)return

self._packetin_icmp_req:发送ICMP_ECHO_REPLY

    def _packetin_icmp_req(self, msg, header_list):# Send ICMP echo reply.in_port = self.ofctl.get_packetin_inport(msg)self.ofctl.send_icmp(in_port, header_list, self.vlan_id,icmp.ICMP_ECHO_REPLY,icmp.ICMP_ECHO_REPLY_CODE,icmp_data=header_list[ICMP].data)srcip = ip_addr_ntoa(header_list[IPV4].src)dstip = ip_addr_ntoa(header_list[IPV4].dst)log_msg = 'Receive ICMP echo request from [%s] to router port [%s].'self.logger.info(log_msg, srcip, dstip, extra=self.sw_id)self.logger.info('Send ICMP echo reply to [%s].', srcip,extra=self.sw_id)

self.packetin_tcp_udp:

响应ICMP_PORT_UNREACH_ERROR

    def _packetin_tcp_udp(self, msg, header_list):# Send ICMP port unreach error.in_port = self.ofctl.get_packetin_inport(msg)self.ofctl.send_icmp(in_port, header_list, self.vlan_id,icmp.ICMP_DEST_UNREACH,icmp.ICMP_PORT_UNREACH_CODE,msg_data=msg.data)srcip = ip_addr_ntoa(header_list[IPV4].src)dstip = ip_addr_ntoa(header_list[IPV4].dst)self.logger.info('Receive TCP/UDP from [%s] to router port [%s].',srcip, dstip, extra=self.sw_id)self.logger.info('Send ICMP destination unreachable to [%s].', srcip,extra=self.sw_id)

9.日志分析

第一个icmp包

[RT][INFO] switch_id=000022a0e427f444: Receive IP packet
from [192.168.1.2] to an internal host [192.168.0.3].
[RT][INFO] switch_id=000022a0e427f444: Send ARP request (flood)

收到icmp包，向所有端口洪泛arp

第一个arp包

[RT][INFO] switch_id=000022a0e427f444: Set implicit routing flow [cookie=0x1]
[RT][INFO] switch_id=000022a0e427f444: Receive ARP reply from [192.168.0.3]to router port [192.168.0.1].
[RT][INFO] switch_id=000022a0e427f444: Send suspend packet to [192.168.0.3].

收到arp包，直接设置点对点的转发规则，然后将上一个icmp包发送到192.168.0.3

返程的报文分析同理

ping一个网关

[RT][INFO] switch_id=000022a0e427f444: Receive ICMP echo request from [192.168.0.3]
to router port [192.168.0.1].
[RT][INFO] switch_id=000022a0e427f444: Send ICMP echo reply to [192.168.0.3].

10.从流量中观察流表下发过程

8,9,10号包？

转发所有的arp报文

其中有个send_arp函数

    def send_arp_request(self, src_ip, dst_ip, in_port=None):# Send ARP request from all ports.for send_port in self.port_data.values():if in_port is None or in_port != send_port.port_no:src_mac = send_port.macdst_mac = mac_lib.BROADCAST_STRarp_target_mac = mac_lib.DONTCARE_STRinport = self.ofctl.dp.ofproto.OFPP_CONTROLLERoutput = send_port.port_noself.ofctl.send_arp(arp.ARP_REQUEST, self.vlan_id,src_mac, dst_mac, src_ip, dst_ip,arp_target_mac, inport, output)

由于连接着OVS的都是veth设备，所以相当于每个veth设备都向自己的对端发送了arp报文，然后如果对端是对应的ip地址的话，对端会返回arp报文。