QEMU/KVM PCI Passthrough(i350) & DPDK 网络性能测试

硬件要求：

CPU必须支持硬件虚拟化（Intel VT-d or AMD-Vi) 和 IOMMU

原图链接

主机配置

设置iommu

IOMMU kernel 配置, 确保以下配置项已启用：

Device Drivers --->[*] IOMMU Hardware Support --->Generic IOMMU Pagetable Support ----[*]   AMD IOMMU support<*>     AMD IOMMU Version 2 driver[*]   Support for Intel IOMMU using DMA Remapping Devices[*]     Support for Shared Virtual Memory with Intel IOMMU[*]     Enable Intel DMA Remapping Devices by default[*]   Support for Interrupt Remapping

IOMMU kernel 启动参数配置：

对于 Intel CPU(VT-d)，使用 intel_iommu=on
对于 AMD CPU(AMD-Vi)，使用 amd_iommu=on

$ sudo uname -a
Linux fc34 5.13.14-200.fc34.x86_64 #1 SMP Fri Sep 3 15:33:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ sudo grubby --update-kernel=ALL --args="iommu=pt intel_iommu=on default_hugepagesz=1G"

reboot 后确认 IOMMU 已启用:

$  dmesg  |grep -i -e DMAR -e IOMMU
[    0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.13.14-200.fc34.x86_64 root=/dev/mapper/fedora_fedora-root ro rd.lvm.lv=fedora_fedora/root iommu=pt intel_iommu=on default_hugepagesz=1G intel_idle.max_cstate=0 processor.max_cstate=0 nohz=on rcu_nocb_poll audit=0 nosoftlockup nospectre_v1 nospectre_v2 spectre_v2=off nospec_store_bypass_disable no_stf_barrier nopti ssbd=off l1tf=off tsx=off tsx_async_abort=off mds=off srbds=off
[    0.013021] ACPI: DMAR 0x000000007EF6A000 0000D8 (v01 INSYDE Romley   00000001 ACPI 00000001)
[    0.013053] ACPI: Reserving DMAR table memory at [mem 0x7ef6a000-0x7ef6a0d7]
[    0.257029] Kernel command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.13.14-200.fc34.x86_64 root=/dev/mapper/fedora_fedora-root ro rd.lvm.lv=fedora_fedora/root iommu=pt intel_iommu=on default_hugepagesz=1G intel_idle.max_cstate=0 processor.max_cstate=0 nohz=on rcu_nocb_poll audit=0 nosoftlockup nospectre_v1 nospectre_v2 spectre_v2=off nospec_store_bypass_disable no_stf_barrier nopti ssbd=off l1tf=off tsx=off tsx_async_abort=off mds=off srbds=off
[    0.257134] DMAR: IOMMU enabled
[    0.469532] DMAR: Host address width 46
[    0.469637] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[    0.469750] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.469886] DMAR: DRHD base: 0x000000ebffc000 flags: 0x1
[    0.469997] DMAR: dmar1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.470132] DMAR: RMRR base: 0x0000007ea8c000 end: 0x0000007ea94fff
[    0.470246] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffe000 IOMMU 0
[    0.470357] DMAR-IR: IOAPIC id 8 under DRHD base  0xebffc000 IOMMU 1
[    0.470481] DMAR-IR: IOAPIC id 9 under DRHD base  0xebffc000 IOMMU 1
[    0.470592] DMAR-IR: HPET id 0 under DRHD base 0xebffc000
[    0.470700] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.471512] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.374638] iommu: Default domain type: Passthrough (set via kernel command line)
[    1.475745] DMAR: No ATSR found
[    1.475855] DMAR: No SATC found
[    1.475862] DMAR: dmar0: Using Queued invalidation
[    1.476206] DMAR: dmar1: Using Queued invalidation
[    1.476483] pci 0000:00:00.0: Adding to iommu group 0
[    1.476660] pci 0000:00:01.0: Adding to iommu group 1

隔离 i350 网卡

确认 kernel 配置中 VFIO 选项都已启用:

Device Drivers ---><M> VFIO Non-Privileged userpsace driver framework --->[*]   VFIO No-IOMMU support ----<M>   VFIO support for PCI devices[*]     VFIO PCI support for VGA devices< >   Mediated device driver framework

注：
fedora 默认不开启 VFIO No-IOMMU support.

配置 hugepages，安装 dpdk 并绑定 i350 网卡到 vfio-pci 驱动:

$ sudo dnf install dpdk dpdk-tools -y
dpdk-20.11-1.fc34.x86_64
dpdk-tools-20.11-1.fc34.x86_64
$ sudo echo 16 > /proc/sys/vm/nr_hugepages
$ sudo modprobe vfio-pci
$ sudo dpdk-hugepages.py -s
Node Pages Size Total
0    8     1Gb    8Gb
1    8     1Gb    8GbHugepages mounted on /dev/hugepages
$ sudo dpdk-devbind.py -b vfio-pci 02:00.3
$ sudo dpdk-devbind.py -sNetwork devices using DPDK-compatible driver
============================================
0000:02:00.3 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=igbNetwork devices using kernel driver
===================================
0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eno1 drv=igb unused=vfio-pci *Active*
0000:05:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp5s0f0 drv=ixgbe unused=vfio-pci
0000:05:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp5s0f1 drv=ixgbe unused=vfio-pci

调整 CPU 模式为 performance 模式：

$ sudo cpupower frequency-set -g performance

QEMU/KVM

由主机PCI 网卡直接分配（PCI Passthrough）:
-device vfio-pci,host=02:00.3
使用主机 hugepages:
-m 8G -object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \ -mem-prealloc -numa node,memdev=mem

$ sudo qemu-kvm -version
QEMU emulator version 3.0.1 (qemu-3.0.1-4.fc29)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers$ sudo qemu-system-x86_64 -M q35,accel=kvm,kernel-irqchip=split \-nographic --enable-kvm \-rtc base=localtime,clock=host \-m 8G -object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \-mem-prealloc -numa node,memdev=mem -overcommit mem-lock=on \-smp 8 -cpu host \-device intel-iommu,intremap=on,device-iotlb=on \-device ioh3420,id=pcie.1,chassis=1 -no-hpet \-netdev tap,id=net0,vhostforce,queues=4 \-device virtio-net-pci,bus=pcie.1,netdev=net0,disable-legacy=on,disable-modern=off,iommu_platform=on,ats=on,vectors=10 \-device vfio-pci,host=02:00.3 \centos8-disk.qcow2

配置虚拟机

kernel 启动参数：

$ sudo grubby --update-kernel=ALL --args="default_hugepagesz=1G"

dpdk & hugepages & vfio 配置：

$ sudo echo 2 > /proc/sys/vm/nr_hugepages
$ sudo modprobe vfio enable_unsafe_noiommu_mode=1
$ sudo modprobe vfio-pci
$ sudo dpdk-devbind -b vfio-pci 00:03.0
$ sudo dpdk-hugepages.py -s
Node Pages Size Total
0    2     1Gb    2GbHugepages mounted on /dev/hugepages$ sudo dpdk-devbind.py  -sNetwork devices using DPDK-compatible driver
============================================
0000:00:03.0 'I350 Gigabit Network Connection 1521' drv=vfio-pci unused=igbNetwork devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1041' if=eth0 drv=virtio-pci unused=vfio-pci *Active*$ sudo cpupower frequency-set -g performance

虚拟机 rxonly 测试

主机 testpmd txonly 发包

$ sudo dpdk-testpmd  -c 0xfc -m 4096 -a 02:00.3 -n 2 \-- -i --rxd=1024 --txd=1024 \--rxq=4 --txq=4 --nb-cores=4 \--forward-mode=txonly \--txpkts=64 \--mlockall -a

testpmd> show port stats all######################## NIC statistics for port 0  ########################RX-packets: 0          RX-missed: 0          RX-bytes:  0RX-errors: 0RX-nombuf:  0         TX-packets: 25246875   TX-errors: 0          TX-bytes:  1615800204Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:      1420592          Tx-bps:    727342616############################################################################
testpmd> quit

虚拟机 testpmd rxonly 收包:

dpdk-testpmd -c 0xfc -m 2048 -a 00:03.0 -- -i -a  \--rxq=1 --txq=1 --nb-cores=4 \--rxd=2048 --txd=2048 --enable-rx-cksum --mlockall \--forward-mode=rxonly

testpmd> show port stats all######################## NIC statistics for port 0  ########################RX-packets: 327892831  RX-missed: 14493499   RX-bytes:  20970135168RX-errors: 0RX-nombuf:  0         TX-packets: 1          TX-errors: 0          TX-bytes:  60Throughput (since last show)Rx-pps:      1420609          Rx-bps:    727351720Tx-pps:            0          Tx-bps:            0############################################################################

虚拟机 txonly 测试

虚拟机 testpmd txonly 发包:

$ sudo dpdk-testpmd -c 0xfc -m 2048 -a 00:03.0 -- -i -a  \--rxq=1 --txq=1 --nb-cores=4 \--rxd=2048 --txd=2048 --enable-rx-cksum --mlockall \--forward-mode=txonly --txpkts=64

testpmd> show port stats  all######################## NIC statistics for port 0  ########################RX-packets: 0          RX-missed: 0          RX-bytes:  0RX-errors: 0RX-nombuf:  0         TX-packets: 5665824    TX-errors: 0          TX-bytes:  362613416Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:      1420607          Tx-bps:    727350144############################################################################
testpmd>

主机 testpmd rxonly 收包：

$ sudo dpdk-testpmd  -c 0xfc -m 4096 -a 02:00.3 -n 2  \-- -i --rxd=2048 --txd=2048 \--rxq=4 --txq=4 --nb-cores=4 \--forward-mode=rxonly --mlockall -a

testpmd> show port stats all######################## NIC statistics for port 0  ########################RX-packets: 34979352   RX-missed: 0          RX-bytes:  2238678868RX-errors: 0RX-nombuf:  0         TX-packets: 0          TX-errors: 0          TX-bytes:  0Throughput (since last show)Rx-pps:      1420598          Rx-bps:    727346200Tx-pps:            0          Tx-bps:            0############################################################################
testpmd>

wirespeed

最终测试结果RX/TX都可达到千兆网络 64 字节小包线速.
CPU:

$ sudo lscpu
[root@centos-guest ~]# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           8
NUMA node(s):        1
Vendor ID:           GenuineIntel
BIOS Vendor ID:      QEMU
CPU family:          6
Model:               45
Model name:          Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
BIOS Model name:     pc-q35-6.1
Stepping:            7
CPU MHz:             2599.998
BogoMIPS:            5199.99
Virtualization:      VT-x
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc archs
[root@centos-guest ~]#

guest rxonly:

testpmd> show port stats all######################## NIC statistics for port 0  ########################RX-packets: 0          RX-missed: 0          RX-bytes:  0RX-errors: 0RX-nombuf:  0         TX-packets: 239624814  TX-errors: 0          TX-bytes:  14377489160Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:      1488316          Tx-bps:    714392640############################################################################
testpmd>

guest txonly:

testpmd> show port stats all######################## NIC statistics for port 0  ########################RX-packets: 125837569  RX-missed: 0          RX-bytes:  7550254396RX-errors: 1RX-nombuf:  0         TX-packets: 0          TX-errors: 0          TX-bytes:  0Throughput (since last show)Rx-pps:      1488207          Rx-bps:    714339808Tx-pps:            0          Tx-bps:            0############################################################################
testpmd>