QEMU/KVM PCI Passthrough(82599ES) & DPDK 网络性能测试

IOMMU & qemu-kvm & hugepages 等具体配置在这里不再重复介绍, 可参考
QEMU/KVM PCI Passthrough(i350) & DPDK 网络性能测试.
这里只上传不同测试场景下性能数据.

原图链接

测试环境

CPU:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          32
On-line CPU(s) list:             0-31
Thread(s) per core:              2
Core(s) per socket:              8
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           45
Model name:                      Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Stepping:                        7
CPU MHz:                         1443.287
CPU max MHz:                     3300.0000
CPU min MHz:                     1200.0000
BogoMIPS:                        5199.80
Virtualization:                  VT-x
L1d cache:                       512 KiB
L1i cache:                       512 KiB
L2 cache:                        4 MiB
L3 cache:                        40 MiB
NUMA node0 CPU(s):               0-7,16-23
NUMA node1 CPU(s):               8-15,24-31
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX vulnerable
Vulnerability Mds:               Vulnerable; SMT vulnerable
Vulnerability Meltdown:          Vulnerable
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:        Vulnerable, IBPB: disabled, STIBP: disabled
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmonpebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear flush_l1d

kernel 启动参数:

[root@fc34 dpdk-wrks]# # grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-5.13.14-200.fc34.x86_64"
args="ro rd.lvm.lv=fedora_fedora/root iommu=pt intel_iommu=on default_hugepagesz=1G intel_idle.max_cstate=0 processor.max_cstate=0 nohz=on rcu_nocb_poll audit=0 nosoftlockup nospectre_v1 nospectre_v2 spectre_v2=off nospec_store_bypass_disable no_stf_barrier nopti ssbd=off l1tf=off tsx=off tsx_async_abort=off mds=off srbds=off $tuned_params isolcpus=1-15"
root="/dev/mapper/fedora_fedora-root"
initrd="/boot/initramfs-5.13.14-200.fc34.x86_64.img $tuned_initrd"
title="Fedora (5.13.14-200.fc34.x86_64) 34 (Server Edition)"
id="3e1208d6a2944886b8c800b19203e82f-5.13.14-200.fc34.x86_64"

NIC:

# lspci  |grep Eth
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
02:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

NIC NUMA:

# cat /sys/bus/pci/devices/0000\:05\:00.0/numa_node
0
# cat /sys/bus/pci/devices/0000\:05\:00.1/numa_node
0

IXIA pktgen

宿主机(host) rxonly performance

#!/bin/sh#ALLOW_DEVS="-a 02:00.3 "
ALLOW_DEVS+="-b 05:00.0 "
ALLOW_DEVS+="-a 05:00.1 "TESTPMD_BIN=/root/devel/dpdk-21.05/v2011-snb/app/dpdk-testpmd
TESTPMD_BIN=dpdk-testpmd
TESTPMD_BIN=/devel/dpdk-21.08/v2108-sandybridge-gcc-11-50mlnx1-1.50218.x86_64/app/dpdk-testpmdARGS_EXTRA=" --huge-unlink "
ARGS_EXTRA+=" --iova-mode=pa "numactl -N 0 -m 0 $TESTPMD_BIN -c 0xfe -m 4096 ${ALLOW_DEVS} -n 2 $ARGS_EXTRA  \-- -i --rxd=2048 --txd=2048 \--rxq=3 --txq=3 --nb-cores=6 \--forward-mode=rxonly \--rss-ip --rss-udp \--numa  --rx-offloads=0 --mbcache=512 --burst=64 \--mlockall -a

vfio with iommu=off intel_iommu=off

测试关闭IOMMU & 开启 vfio enable_unsafe_noiommu_mode=1 RX 性能:

testpmd> show port stats 1######################## NIC statistics for port 1  ########################RX-packets: 266636840  RX-missed: 0          RX-bytes:  15998212086RX-errors: 0RX-nombuf:  0         TX-packets: 0          TX-errors: 0          TX-bytes:  0Throughput (since last show)Rx-pps:     14881458          Rx-bps:   7143102968Tx-pps:            0          Tx-bps:            0############################################################################
testpmd>

vfio with iommu=on intel_iommu=on

测试开启IOMMU & 开启 vfio RX 性能:

testpmd> show port stats 1######################## NIC statistics for port 1  ########################RX-packets: 60085340   RX-missed: 151041955  RX-bytes:  3605124726RX-errors: 0RX-nombuf:  0         TX-packets: 0          TX-errors: 0          TX-bytes:  0Throughput (since last show)Rx-pps:      4087223          Rx-bps:   1961860888Tx-pps:            0          Tx-bps:            0############################################################################
testpmd>

igb_uio

测试使用 igb_uio RX 性能:

testpmd> show port stats 1######################## NIC statistics for port 1  ########################RX-packets: 70484884   RX-missed: 0          RX-bytes:  4229094726RX-errors: 0RX-nombuf:  0         TX-packets: 0          TX-errors: 0          TX-bytes:  0Throughput (since last show)Rx-pps:     14881470          Rx-bps:   7143108336Tx-pps:            0          Tx-bps:            0############################################################################
testpmd>

宿主机(host) txonly performance

#!/bin/sh#ALLOW_DEVS="-a 02:00.3"
#ALLOW_DEVS="-a 02:10.3"
ALLOW_DEVS+="-a 05:00.0 "
ALLOW_DEVS+="-a 05:00.1 "TESTPMD_BIN=dpdk-testpmd
TESTPMD_BIN=/root/devel/dpdk-21.05/v2011-snb/app/dpdk-testpmd
TESTPMD_BIN=/devel/dpdk-21.08/v2108-sandybridge-gcc-11-50mlnx1-1.50218.x86_64/app/dpdk-testpmdARGS_EXTRA=" --iova-mode=pa --huge-unlink"numactl -N 0 -m 0 $TESTPMD_BIN -c 0xfc -m 4096 ${ALLOW_DEVS} -n 2 $ARGS_EXTRA  \-- -i --rxd=2048 --txd=2048 \--rxq=1 --txq=1 --nb-cores=4 \--forward-mode=txonly --txpkts=60 \--numa  --txonly-multi-flow --tx-offloads=0 --mbcache=512 --burst=64 \--mlockall -a

vfio-pci with iommu=off intel_iommu=off

测试关闭 IOMMU & vfio enable_unsafe_noiommu_mode=1 TX 性能:

testpmd> show  port stats 1######################## NIC statistics for port 1  ########################RX-packets: 1          RX-missed: 0          RX-bytes:  66RX-errors: 0RX-nombuf:  0         TX-packets: 165155071  TX-errors: 0          TX-bytes:  9909301124Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:     14880763          Tx-bps:   7142767328############################################################################
testpmd>

vfio-pci with iommu=on intel_iommu=on

测试开启 IOMMU TX 性能:

testpmd> show port stats 1######################## NIC statistics for port 1  ########################RX-packets: 1          RX-missed: 0          RX-bytes:  66RX-errors: 0RX-nombuf:  0         TX-packets: 42082664   TX-errors: 0          TX-bytes:  2524958752Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:      4163606          Tx-bps:   1998532944############################################################################
testpmd>

igb_uio

测试使用 igb_uio TX 性能:

testpmd>
testpmd> show  port stats  all######################## NIC statistics for port 0  ########################RX-packets: 1          RX-missed: 0          RX-bytes:  66RX-errors: 0RX-nombuf:  0         TX-packets: 218982941  TX-errors: 0          TX-bytes:  13138973388Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:     14880768          Tx-bps:   7142768832############################################################################
testpmd>

QEMU/KVM 虚拟机(guest) rxonly performance

iommu=off intel_iommu=off

测试关闭 IOMMU & vfio enable_unsafe_noiommu_mode=1 RX 性能:

testpmd> show port stats all######################## NIC statistics for port 0  ########################RX-packets: 45891304   RX-missed: 110802769  RX-bytes:  2753482566RX-errors: 0RX-nombuf:  0         TX-packets: 1          TX-errors: 0          TX-bytes:  66Throughput (since last show)Rx-pps:      4305963          Rx-bps:   2066854672Tx-pps:            0          Tx-bps:            0############################################################################
testpmd>

iommu=pt intel_iommu=on

测试开启 IOMMU RX 性能:

testpmd> show port stats  all######################## NIC statistics for port 0  ########################RX-packets: 23093640   RX-missed: 54535991   RX-bytes:  1385622966RX-errors: 0RX-nombuf:  0         TX-packets: 1          TX-errors: 0          TX-bytes:  66Throughput (since last show)Rx-pps:      4292522          Rx-bps:   2060408272Tx-pps:            0          Tx-bps:            0############################################################################
testpmd>

QEMU/KVM 虚拟机(guset) txonly performance

iommu=off intel_iommu=off

测试关闭 IOMMU & vfio enable_unsafe_noiommu_mode=1 TX 性能:

testpmd> show port stats all######################## NIC statistics for port 0  ########################RX-packets: 1          RX-missed: 0          RX-bytes:  66RX-errors: 0RX-nombuf:  0         TX-packets: 64208444   TX-errors: 0          TX-bytes:  3852506064Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:      4264733          Tx-bps:   2047075928############################################################################
testpmd>

iommu=pt intel_iommu=on

测试开启 IOMMU TX 性能:

testpmd> show port stats  all######################## NIC statistics for port 0  ########################RX-packets: 1          RX-missed: 0          RX-bytes:  66RX-errors: 0RX-nombuf:  0         TX-packets: 38054588   TX-errors: 0          TX-bytes:  2435493360Throughput (since last show)Rx-pps:            0          Rx-bps:            0Tx-pps:      4238879          Tx-bps:   2170316088############################################################################
testpmd> quit

结论

intel 82599ES 这款10G网卡在关闭iommu模式下单核 RX&TX 在宿主机(host)均可达到小包线速.
开启iommu模式下小包性能不是很理想, 具体原因待查.

后续1:
检查内核日志发现 DMAR 报错 DMAR: DRHD: handling fault status reg , 估计是这个原因导致 82599ES 开启IOMMU后性能降低, 具体原因待查.

# dmesg  |grep "IOMMU\|DMAR"
[    0.013139] ACPI: DMAR 0x000000007EF6A000 0000D8 (v01 INSYDE Romley   00000001 ACPI 00000001)
[    0.013170] ACPI: Reserving DMAR table memory at [mem 0x7ef6a000-0x7ef6a0d7]
[    0.257528] DMAR: IOMMU enabled
[    0.469389] DMAR: Host address width 46
[    0.469494] DMAR: DRHD base: 0x000000fbffe000 flags: 0x0
[    0.469607] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.469742] DMAR: DRHD base: 0x000000ebffc000 flags: 0x1
[    0.469853] DMAR: dmar1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.469988] DMAR: RMRR base: 0x0000007ea8c000 end: 0x0000007ea94fff
[    0.470101] DMAR-IR: IOAPIC id 10 under DRHD base  0xfbffe000 IOMMU 0
[    0.470212] DMAR-IR: IOAPIC id 8 under DRHD base  0xebffc000 IOMMU 1
[    0.470337] DMAR-IR: IOAPIC id 9 under DRHD base  0xebffc000 IOMMU 1
[    0.470447] DMAR-IR: HPET id 0 under DRHD base 0xebffc000
[    0.470556] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.471342] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    1.255591] DMAR: No ATSR found
[    1.255700] DMAR: No SATC found
[    1.255707] DMAR: dmar0: Using Queued invalidation
[    1.256050] DMAR: dmar1: Using Queued invalidation
[    1.338371] DMAR: Intel(R) Virtualization Technology for Directed I/O
[  366.127902] DMAR: DRHD: handling fault status reg 2
[  366.127914] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set
[  366.135408] DMAR: DRHD: handling fault status reg 102
[  366.135422] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c7fb000 [fault reason 06] PTE Read access is not set
[  396.401306] DMAR: DRHD: handling fault status reg 202
[  396.401315] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set
[  396.408879] DMAR: DRHD: handling fault status reg 302
[  396.408884] DMAR: [DMA Read] Request device [05:00.1] PASID ffffffff fault addr 25c813000 [fault reason 06] PTE Read access is not set

后续2:
重新加载默认BIOS设置后不再报DMAR 报错 DMAR: DRHD: handling fault status reg错误, 但是开启IOMMU后 82599ES 性能依然无法提升, 不知是否跟CPU有关系.

find /sys/kernel/iommu_groups/ -type l |grep 05\:
/sys/kernel/iommu_groups/25/devices/0000:05:00.1
/sys/kernel/iommu_groups/24/devices/0000:05:00.0

DPDK 19.11 Intel NIC Performance Report 在Intel® Xeon® Processor Platinum 8180 (38.5 M Cache, 2.50 GHz) CPU下无此问题.