使用libvirt和qemu将pci pass through设备添加到虚拟机上

透传的优势

guest使用透传设备可以获得设备近乎原生的性能,

PCI pass-throught设备给动态迁移带来的问题, dest host可能没有同样的硬件.

就算可以模拟一个设备,但是原始设备的内部状态不能获得.

VT-d support

In order to assign devices in KVM, you’ll need a system which supports VT-d. This has nothing to do with the VT-x support of your CPU, VT-d needs to be supported by both your chipset on your motherboard and by your CPU.

If you are in doubt whether your motherboard or CPU supports VT-d or not, the Xen VT-d wikipage has some pointers of VT-d enabled chipsets, motherboards and CPUs: http://wiki.xensource.com/xenwiki/VTdHowTo

If your hardware doesn’t have an IOMMU (“Intel VT-d” support in case of Intel – “AMD I/O Virtualization Technology” support in case of AMD), you’ll not be able to assign devices in KVM. Some work towards allowing this were done, but the code never made it into KVM, due to various issues with the code. At the moment it doesn’t seem like device assignment without hardware support, will ever be integrated into KVM.

Assignment of graphics cards are not officially supported at the moment, but there has been some success passing through a secondary Radeon HD 5850 as a VM’s secondary display.

资料:http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

在host选择PCI pass through设备

reboot and verify that your system has IOMMU support

AMD Machine

dmesg | grep AMD-Vi

...

AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40

AMD-Vi: Lazy IO/TLB flushing enabled

AMD-Vi: Initialized for Passthrough Mode

...

Intel Machine

dmesg | grep -e DMAR -e IOMMU

...

DMAR:DRHD base: 0x000000feb03000 flags: 0x0

IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000

...

If you get no output you’ll need to fix this before moving on. Check if your hardware supports VT-d and check that it has been enabled in BIOS.

NOTE: If you still get an error “No IOMMU found.” Check dmesg for errors suggesting your BIOS is broken. Another possible reason: CONFIG_DMAR_DEFAULT_ON is not set. In that case, pass “intel_iommu=on” as kernel parameter to enable it. AMD uses different kernel parameter than Intel, on AMD you need to pass “iommu=pt iommu=1”.

请看附录:No IOMMU found 解决

选择要使用的透传设备

# lspci -nn

00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0100] (rev 09)

......

00:1b.0 Audio device [0403]: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller [8086:1c20] (rev 04)

.....

00:1f.3 SMBus [0c05]: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller [8086:1c22] (rev 04)

友情提示:使用透传设备时,拿USB控制器作实验,可能鼠标键盘不能使用.请谨慎.

将设备从宿主机上解除绑定

使用echo命令,将设备从host机器上解除绑定,将来用于guest机器. For example:

echo "8086 1c20" > /sys/bus/pci/drivers/pci-stub/new_id

echo 0000:00:1b.0 > /sys/bus/pci/devices/0000:00:1b.0/driver/unbind

echo 0000:00:1b.0 > /sys/bus/pci/drivers/pci-stub/bind

关闭虚拟机

关闭虚拟机,修改配置文件.

使用libvirt进行pci pass through

修改虚拟机配置文件

<devices> …

<hostdev mode='subsystem' type='pci' managed='yes'>

<source>

<address domain='0x000' bus='0x00′ slot='0x1b' function='0x0' />

</source>

</hostdev>

</devices>

在修改完虚拟机配置文件后,运行虚拟机.

使用qemu进行pci pass through

使用qemu进行pci pass through也是一样的简单的.我们需要一个已经创建好操作系统的虚拟机.

在qemu命令行运行以下命令以启动虚拟机

/usr/bin/qemu-kvm-name vdisk -enable-kvm -m 512 -smp 2 \

-hda/mnt/nfs/vdisk.img \

-monitor stdio \

-vnc 0.0.0.0:0 \

-device pci-assign,host=00:1b.0

这样就将设备挂载到虚拟机上了.

参数-device pci-assign,host=00:1b.0说的是使用一个pci设备,并提供一个设备的地址.
参数-monitor stdio是使用一个标准的控制台输出.在命令行中进行输入命令,等等迁移的时候也在这里输入命令.

附录1:No IOMMU found 解决

启动虚拟机的时候出现了iommu的问题.以供大家参考

1.查看错误日志说明

在配置好XML文件后,启动虚拟机,遇到一个问题.

error: Failed to start domain vdisk
error: Unable to read from monitor: Connection reset by peer

查看虚拟机日志( cat /var/log/libvirt/qemu/vdisk.log )信息.

char device redirected to /dev/pts/3
No IOMMU found. Unable to assign device "hostdev0"
qemu-system-x86_64: -device pci-assign,configfd=20,host=00:1b.0,id=hostdev0,bus=pci.0,addr=0x4: Device 'pci-assign' could not be initialized
2013-07-08 06:41:23.256+0000: shutting down

上网查阅资料,说是要在BIOS上设置虚拟化,然后在引导程序里也要设置iommu.可以查看一下自己的电脑信息是否开启了.

2.查看信息gurb的引导信息

1 2	`# cat /proc/cmdline` `BOOT_IMAGE=/vmlinuz-3.9.6-200.fc18.x86_64 root=/dev/mapper/fedora-home ro rd.lvm.lv=fedora/swap rd.md=0 rd.dm=0 rd.lvm.lv=fedora/home rd.luks=0 vconsole.keymap=us rhgb quiet LANG=en_US.UTF-8`

可以发现,我的还未开启intel_iommu=on选项.所以接下来我们来激活它.

3.激活intel_iommu=on

Activate Intel VT-d in the kernel

Activate Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line of the kernel line in the/boot/grub/grub.conf file. The example below is a modified grub.conf file with Intel VT-d activated.

对于intel的cpu和amd的cpu,在grub配置上是不同的,具体的配置请参考文章:http://pve.proxmox.com/wiki/Pci_passthrough

4.更新grub

在编辑完grub文件后,需要更新

1 2	`grub2-mkconfig # fedora arch centos` `update-grub # ubuntu debian`

5.重启电脑,使其生效

1 2	`# cat /proc/cmdline` `BOOT_IMAGE=/vmlinuz-3.9.6-200.fc18.x86_64 root=/dev/mapper/fedora-home ro rd.lvm.lv=fedora/swap rd.md=0 rd.dm=0 rd.lvm.lv=fedora/home rd.luks=0 vconsole.keymap=us rhgb quiet intel_iommu=on LANG=en_US.UTF-8`

发现开机已经启动了inte _iommu=on了.再次启动虚拟机已经就不会出现这个bug了.

附录2 PCI pass through 失败要关闭SELinux

我用的是Fedora 18 ,将SELinux给disalbed了,但是发现PCI pass through出先问题.上网看到文章PCI passthrough fails in qemu-kvm unless selinux is disabled 说,要将selinux设置成permissive模式,于是这个问题解决了.

参考资料

几句话科普一下虚拟化，VT-x，VT-d http://bbs.pceva.com.cn/thread-55077-1-1.html
Wikipedia – x86 virtualization http://en.wikipedia.org/wiki/X86_virtualization
Red Hat Enterprise Linux Virtualization Guide – PCI passthrough http://www.linuxtopia.org/online_books/rhel6/rhel_6_virtualization/rhel_6_virtualization_chap-Virtualization-PCI_passthrough.html
Adding pass-through devices to guests with libvirt http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatlibvirtaddpassdev.htm
SELinux 入门 http://linuxtoy.org/archives/selinux-introduction.html

声明：未经允许禁止转载东东东陈煜东的博客文章，谢谢。如经授权，转载请注明：转载自东东东陈煜东的博客

本文链接地址: 使用libvirt和qemu将pci pass through设备添加到虚拟机上 – https://www.chenyudong.com/archives/add-pci-pass-through-device-to-guest-vm-with-libvirt-and-qemu.html