rust-vmm 学习
V0.1.0 feature
base knowledge:
Architecture of the Kernel-based Virtual Machine (KVM)
用rust-vmm打造未来的虚拟化架构
KVM内核文档阅读笔记
<Mastering KVM Virtualization>:第二章 KVM内部原理
Using the KVM API (org)
Example for a simple vmm (github, search pub fn test_vm())
kvm-ioctls (github)
run test:
cd vmmFILE_LINE=`git grep -n "pub fn test_vm"` && echo ${FILE_LINE%:*}cargo test test_vm
FILE_LINE=`git grep -n "pub fn test_vm"` && echo ${FILE_LINE%:*}src/vm.rs:854cargo test test_vm |grep "Running target/debug/deps/"Finished dev [unoptimized + debuginfo] target(s) in 0.16sRunning target/debug/deps/vmm-5f7f458cb725298f
rust-gdb target/debug/deps/vmm-5f7f458cb725298f
(gdb) info functions test_vm
All functions matching regular expression "test_vm":
(gdb) help info
(gdb) info sources
(gdb) b src/vm.rs:854
No source file named src/vm.rs.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) run --test test_vm
# note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.RUST_BACKTRACE=1 cargo runvim ./cloud-hypervisor/src/main.rs:ConqueGdbExe rust-gdb :ConqueGdb target/debug/cloud-hypervisor
(gdb) info functions main
All functions matching regular expression "main":
File cloud-hypervisor/src/main.rs:
static void cloud_hypervisor::main::h9d806d3576285a29(void);
(gdb) b cloud_hypervisor::main::h9d806d3576285a29
(gdb) run --kernel ./hypervisor-fw \--disk ./clear-29160-kvm.img \--cpus 4 \--memory 512 \--net "tap=,mac=,ip=,mask=" \--rng(gdb) run target/debug/cloud-hypervisor \--kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \--disk ./clear-29160-kvm.img \--cmdline "console=ttyS0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3" \--cpus 4 \--memory 512 \--net "tap=,mac=,ip=,mask=" \--rng
Example: debug a Virtio devices(Rng)
sudo ~/.cargo/bin/rust-gdb cloud-hypervisor/target/debug/cloud-hypervisor (gdb) i functions Rng::new File vm-virtio/src/rng.rs: struct Result<vm_virtio::rng::Rng, std::io::error::Error> vm_virtio::rng::Rng::new::h79792d122294607e(struct &str); (gdb) b vm_virtio::rng::Rng::new::h79792d122294607e Breakpoint 1 at 0x6330e4: file vm-virtio/src/rng.rs, line 156. (gdb) r --kernel ./linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \--disk ./clear-29160-kvm.img \--cmdline "console=ttyS0 reboot=k panic=1 nomodules i8042.noaux i8042.nomux i8042.nopnp i8042.dumbkbd root=/dev/vda3" \--cpus 4 \--memory 512 \--net "tap=,mac=,ip=,mask=" \--rng (gdb) bt #0 vm_virtio::rng::Rng::new::h79792d122294607e (path="/dev/urandom") at vm-virtio/src/rng.rs:156 #1 0x0000555555ac81fa in vmm::vm::DeviceManager::new::hb95c517461758006 (memory=GuestMemoryMmap = {...}, allocator=0x7fffffff65e0, vm_fd=0x7fffffff59f0, vm_cfg=0x7fffffff7a18, msi_capable=true, userspace_ioapic=true) at vmm/src/vm.rs:572 #2 0x0000555555acd235 in vmm::vm::Vm::new::h319ad49fd3f83723 (kvm=0x7fffffff731c, config=VmConfig = {...}) at vmm/src/vm.rs:905 #3 0x0000555555aafed1 in vmm::boot_kernel::hc06685bd1a4087c9 (config=VmConfig = {...}) at vmm/src/lib.rs:59 #4 0x00005555556d377b in cloud_hypervisor::main::hc6d3862453504aa4 () at src/main.rs:114 #5 0x00005555556d0110 in std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hb71d053f091b9e72 () at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/rt.rs:64 #6 0x0000555555c36943 in {{closure}} () at src/libstd/rt.rs:49 #7 do_call<closure,i32> () at src/libstd/panicking.rs:297 #8 0x0000555555c3a0ca in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:87 #9 0x0000555555c3744d in try<i32,closure> () at src/libstd/panicking.rs:276 #10 catch_unwind<closure,i32> () at src/libstd/panic.rs:388 #11 lang_start_internal () at src/libstd/rt.rs:48 #12 0x00005555556d00e9 in std::rt::lang_start::hd3791e31e26e1c55 (main=0x5555556d2720 <cloud_hypervisor::main::hc6d3862453504aa4>, argc=14, argv=0x7fffffffe1a8)at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/rt.rs:64 #13 0x00005555556d3bca in main () (gdb) set print pretty on
(gdb) info functions Rng.*activate (gdb) info functions Rng.*VirtioDevice.*activate All functions matching regular expression "Rng.*VirtioDevice.*activate":File vm-virtio/src/rng.rs: struct Result<(), vm_virtio::ActivateError> _$LT$vm_virtio..rng..Rng$u20$as$u20$vm_virtio..device..VirtioDevice$GT$::activate::hdedcc2490e8a5155(struct Rng *, struct GuestMemoryMmap, struct Arc<alloc::boxed::Box<Fn<(&vm_virtio::queue::Queue)>>>, struct Arc<core::sync::atomic::AtomicUsize>, struct Vec<vm_virtio::queue::Queue>, struct Vec<vmm_sys_util::eventfd::EventFd>); (gdb) b _$LT$vm_virtio..rng..Rng$u20$as$u20$vm_virtio..device..VirtioDevice$GT$::activate::hdedcc2490e8a5155(gdb) b vm-virtio/src/rng.rs:236 (gdb) c Thread 2 "cloud-hyperviso" hit Breakpoint 2, _$LT$vm_virtio..rng..Rng$u20$as$u20$vm_virtio..device..VirtioDevice$GT$::activate::hdedcc2490e8a5155 (self=0x555555f71f10, mem=GuestMemoryMmap = {...}, interrupt_cb=Arc<alloc::boxed::Box<Fn<(&vm_virtio::queue::Queue)>>> = {...}, status=Arc<core::sync::atomic::AtomicUsize> = {...}, queues=Vec<vm_virtio::queue::Queue>(len: 1, cap: 1) = {...}, queue_evts=Vec<vmm_sys_util::eventfd::EventFd>(len: 1, cap: 1) = {...}) at vm-virtio/src/rng.rs:236 236 if queues.len() != NUM_QUEUES || queue_evts.len() != NUM_QUEUES { (gdb) bt #0 _$LT$vm_virtio..rng..Rng$u20$as$u20$vm_virtio..device..VirtioDevice$GT$::activate::hdedcc2490e8a5155 (self=0x555555f71f10, mem=GuestMemoryMmap = {...}, interrupt_cb=Arc<alloc::boxed::Box<Fn<(&vm_virtio::queue::Queue)>>> = {...}, status=Arc<core::sync::atomic::AtomicUsize> = {...}, queues=Vec<vm_virtio::queue::Queue>(len: 1, cap: 1) = {...}, queue_evts=Vec<vmm_sys_util::eventfd::EventFd>(len: 1, cap: 1) = {...}) at vm-virtio/src/rng.rs:236 #1 0x0000555555b84dc5 in _$LT$vm_virtio..transport..pci_device..VirtioPciDevice$u20$as$u20$pci..device..PciDevice$GT$::write_bar::h7ff0b55cec4a3130 (self=0x555555f72170, offset=20, data=&[u8](len: 1) = {...}) at vm-virtio/src/transport/pci_device.rs:549 #2 0x0000555555b85692 in _$LT$vm_virtio..transport..pci_device..VirtioPciDevice$u20$as$u20$devices..bus..BusDevice$GT$::write::h32bdbeb4a3c402c0 (self=0x555555f72170, offset=20, data=&[u8](len: 1) = {...}) at vm-virtio/src/transport/pci_device.rs:590 #3 0x0000555555be16ab in devices::bus::Bus::write::h23c30f9081f573b2 (self=0x7fffd6ec2788, addr=68717891604, data=&[u8](len: 1) = {...}) at devices/src/bus.rs:158 #4 0x0000555555ac561b in vmm::vm::Vcpu::run::h523fa30e0aaa7af6 (self=0x7fffd6ec2760) at vmm/src/vm.rs:363 #5 0x0000555555ae8eb3 in vmm::vm::Vm::start::_$u7b$$u7b$closure$u7d$$u7d$::h643431f00d7471fd () at vmm/src/vm.rs:1110 #6 0x0000555555ad9f12 in std::sys_common::backtrace::__rust_begin_short_backtrace::hd688e526ad581234 (f=...)at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/sys_common/backtrace.rs:135 #7 0x0000555555ad7931 in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h8d7f8a62b8d1257c ()at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/thread/mod.rs:469 #8 0x0000555555af14a1 in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h1f5925e04de15538 (self=..., _args=0) at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/panic.rs:309 #9 0x0000555555b00eda in std::panicking::try::do_call::hebfb4f2c896f9302 (data=0x7fffd6ec2a40 "\020)\367UUU")at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/panicking.rs:297 #10 0x0000555555c3a0ca in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:87 #11 0x0000555555b00ca0 in std::panicking::try::h6da33e6171d79492 (f=...) at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/panicking.rs:276 #12 0x0000555555af15d3 in std::panic::catch_unwind::h1660cc8af66e7984 (f=...) at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/panic.rs:388 #13 0x0000555555ad7336 in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h85cc8e716556c7d2 ()at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/libstd/thread/mod.rs:468 #14 0x0000555555ad7c7b in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h5eac52d3a365b1f5 (self=0x555555f72d70, args=0)at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/liballoc/boxed.rs:749 #15 0x0000555555c3985e in call_once<(),()> () at /rustc/91856ed52c58aa5ba66a015354d1cc69e9779bdf/src/liballoc/boxed.rs:759 #16 start_thread () at src/libstd/sys_common/thread.rs:14 #17 thread_start () at src/libstd/sys/unix/thread.rs:81 #18 0x00007ffff77b56ba in start_thread (arg=0x7fffd6ec4700) at pthread_create.c:333 #19 0x00007ffff72d541d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
GDB commands by function
Print settings
set print pretty on
virtio (Github)
virtqueue的每个部分的内存对齐和大小要求如下表,其中Queue Size对应virtqueue中的最大buffer数,始终为2的n次幂,以特定于总线的方式指定:
Driver将sk_buffer填充进scatterlist table中(只是设置地址没有数据搬移),然后通过计算得到GPA并将GPA写入Descriptor Table中,同时将Desc chain的head记录到Available Ring中,然后通过PIO的方式通知Device,Device发包并更新Used Ring。
virtio 系列文章
Virtio原理简介
6.1.Virtio 原理与Guest OS driver
Virtio概述和基本原理(KVM半虚拟化驱动)
virtio基本原理(kvm半虚拟化驱动)
VIRTIO VRING工作机制分析
VirtIO 代码分析(rust-vmm) cloud-hypervisor Latest commit f9036a1
Test_vm
vmm/src/vm.rs::test_vm是一个很好的入门例程。
VM创建过程:
1. src/main.rs 的main函数, 解析参数,然后boot_kernel (vmm/src/lib.rs).
2. 创建kvm (kvm_ioctls crate, 打开/dev/kvm得到fd),
3. vmm/src/vm.rs 创建Vm,
(kvm.create_vm() 得到vm的fd),
初始化guest的memory,
CPU的Vec,
DeviceManager::new 来创建设备, 各种设备,包括PCI设备在此创建。
4. boot_kernel中,先load_kernel, 最后start (创建期望的个数的Vcpu,返回fd, 放在Vec<thread::JoinHandle<()>>中).
创建Vcpu时,会传入DeviceManager的各种信息:
io_bus = self.devices.io_bus.clone()
mmio_bus = self.devices.mmio_bus.clone()
ioapic = if let Some(ioapic) = &self.devices.ioapic {Some(ioapic.clone())}
io_bus和mmio_bus都是devices/src/bus.rs::Bus类型 {devices: BTreeMap<BusRange, Arc<Mutex<BusDevice>>>}
然后在每个thread中, vcpu.run (ioctls/vcpu.rs pub fn run).
5. 最后,control_loop 通过epoll从stdin读入数据,发给VM的serial.
PCI设备创建过程(Rng为例):
1. vmm/src/vm.rs: DeviceManager会调用Rng::new 生成virtio_rng_device
2. DeviceManager::add_virtio_pci_device将virtio_rng_device 生成一个VirtioPciDevice,并加入到pci_root(PciRoot::new)中
PciConfigIo::new(pci_root)
还会调用PciConfigIo::register_mapping 设置bus, 会将 (address, size), VirtioPciDevice插入到bus 的map中。
3. VirtioPciDevice::new 会设置
PCI configuration registers
virtio PCI common configuration
MSI-X config
PCI interrupts
virtio queues, 会Rng转化为queue,并设置queue_evts(EventFd::new), vm_fd.register_ioevent
Guest memory
Setting PCI BAR
PCI设备访问过程(Rng为例mmio为例):
1. 每个VCPU线程vmm::vm::Vcpu::run会判断VCPU的退出情况。以MmioWrite为例
2. devices/src/bus.rs::write 会遍历找到对应的device 进行 write_bar
3. 调用device.activate, 即Rng::activate, 会在一个线程中调用RngEpollHandler::run
4. process_queue
virtIO相关结构
1. struct Descriptor
/// A virtio descriptor constraints with C representive
struct Descriptor {
addr: u64,
len: u32,
flags: u16,
next: u16,
}
2. pub struct DescriptorChain
/// A virtio descriptor chain. 包含了Descriptor的4个字段
此外还包括申请的mmap,
desc_table的地址
desc table的索引
1 /// A virtio descriptor chain. 2 pub struct DescriptorChain<'a> { 3 mem: &'a GuestMemoryMmap, 4 desc_table: GuestAddress, 5 queue_size: u16, 6 ttl: u16, // used to prevent infinite chain cycles 7 8 /// Index into the descriptor table 9 pub index: u16, 10 11 /// Guest physical address of device specific data 12 pub addr: GuestAddress, 13 14 /// Length of device specific data 15 pub len: u32, 16 17 /// Includes next, write, and indirect bits 18 pub flags: u16, 19 20 /// Index into the descriptor table of the next descriptor if flags has 21 /// the next bit set 22 pub next: u16, 23 }
3. AvailIter
/// Consuming iterator over all available descriptor chain heads in the queue.
1 /// Consuming iterator over all available descriptor chain heads in the queue. 2 pub struct AvailIter<'a, 'b> { 3 mem: &'a GuestMemoryMmap, 4 desc_table: GuestAddress, 5 avail_ring: GuestAddress, 6 next_index: Wrapping<u16>, 7 last_index: Wrapping<u16>, 8 queue_size: u16, 9 next_avail: &'b mut Wrapping<u16>, 10 }
View Code
4. Queue
/// A virtio queue's parameters. 包含了desc_table, avail_ring, used_ring 和中断向量的索引。
1 /// A virtio queue's parameters. 2 pub struct Queue { 3 /// The maximal size in elements offered by the device 4 max_size: u16, 5 6 /// The queue size in elements the driver selected 7 pub size: u16, 8 9 /// Inidcates if the queue is finished with configuration 10 pub ready: bool, 11 12 /// Interrupt vector index of the queue 13 pub vector: u16, 14 15 /// Guest physical address of the descriptor table 16 pub desc_table: GuestAddress, 17 18 /// Guest physical address of the available ring 19 pub avail_ring: GuestAddress, 20 21 /// Guest physical address of the used ring 22 pub used_ring: GuestAddress, 23 24 next_avail: Wrapping<u16>, 25 next_used: Wrapping<u16>, 26 }
View Code
5. Rng
Rng 是不包括Queue字段的
VirtioPciDevice 的new时候,生成Queue, 还会生成bar的地址fn allocate_bars。
Queue传入的地址就是为VM申请的guest_memory
VirtioPciDevice::read_bar 和 VirtioPciDevice::write_bar (VirtioPciDevice被插入了bus中) 会调用common_config: VirtioPciCommonConfig的write 和read Queue的各个field.
PCI bar地址
1. vm new的时候,会初始化一个allocator = SystemAllocator::new.
2. 这个allocator会传给device_manager = DeviceManager::new .
3. DeviceManager new的时候,在add_virtio_pci_device的时候,会分配bars = virtio_pci_device.allocate_bars
4. 具体由vm-allocator/src/address.rs::AddressAllocator 的 AddressAllocator::allocate分配bar地址
5. SystemAllocator::allocate_mmio_addresses 会 Reserves a section of `size` bytes of MMIO address space.
PCI passtrough代码分析(从5372554开始支持VFIO)
REF:
VFIO ——将 DMA 映射暴露给用户态 (推荐)
vfio-mdev逻辑空间分析 (mdev设备也是基于VFIO)
地址空间的故事 (介绍了CCIX)
VFIO driver (VFIO的ORG文档, 内核及用户空间源码,内核 3.6)
VFIO简介 转自developerworks(很多不错的文章)
FPGA通过vfio实现pci-passthrough给kvm虚拟机
Linux iommu和vfio概念空间解构
Cloud Phypervisor的实现
1. 5372554 实现了C语言的vfio-bindings
The default bindings are generated from the 5.0.0 Linux userspace API.
2. 2cec3aa 实现了VFIO API (用户态的调用这些API,操作VFIO, VFIO ——将 DMA 映射暴露给用户态 )
实现了 setup_dma_map, enable_msi, enable_msix, region_read, region_write
VFIO平台无关的接口层
- 向用户态提供访问硬件设备的接口
- 向用户态提供配置IOMMU的接
3. db5b476 实现了VFIO PCI device 的各种操作。VFIO实现层(PCI设备实现层)
allocate bar
write/read config register 会read/write region (write_config_register)
write/read bars 也会read/write region (write_bar)
allocate bar 时会 setup_dma_map
4. 20f0116 实现了MSI和MSI-X的设置
vfio: pci: Track MSI and MSI-X capabilities
5. c93d536 设置中断(需要研究一下原理)
vfio: pci: Build the KVM routes
VFIO实现层(PCI设备实现层)
6. b746dd7
7. 4d16ca8 实现了PCI passthroug (Doc)
vmm: Support direct device assignment
5ae3144 testcase
8. 927861c .. fa41ddd VFIO的bug fix.
转载于:https://www.cnblogs.com/shaohef/p/11025893.html
rust-vmm 学习相关推荐
- rust - trait学习
通过对<Rust 程序设计语言>,<通过例子学 Rust 中文版>以及令狐一冲老师对相关知识点的学习总结而成. rust - trait学习 1 trait 的基本含义 2 定 ...
- Rust太难?那是你没看到这套Rust语言学习万字指南!
摘要:从开发环境.语法.属性.内存管理和Unicode等五部分,为你带来一份详细的Rust语言学习的精华总结内容. 一.Rust开发环境指南 1.1 Rust代码执行 根据编译原理知识,编译器不是直接 ...
- Rust语言学习大纲
Time: 20190919 接下来,我们将循着这个足迹来学习了解Rust语言,实战Rust语言,希望对你也有一定的帮助~ 2019.10 Update: 第一届PAT算法直播课培训班招募帖,欢迎点击 ...
- Rust Reqwest 学习
reqwest 0.11.4 学习 以应用为主, 在实践中体验 rust 的强大和快捷. 首先以 Web 开发为主题,学习一个强大的 HTTP 客户端库 reqwest . Crate reqwest ...
- Rust 闭包学习 (Fn/FnMut/FnOnce)
文章目录 闭包作为参数 闭包作为结构体属性 异步使用闭包 项目地址 学习 Rust 闭包记录 闭包作为参数 fn main() {parameter_is_callback_t(Box::new(|| ...
- Rust结伴学习 — 前言
前言 本系列旨在记录自己学习Rust的过程,也希望借此方式激励自己能够坚持下去,内容可能比较简单,希望大家见谅,也希望大家可以督促我坚持下去~~ 为什么学习Rust 没有很特别的原因,只是因为Rust ...
- rust入门学习指南
背景介绍 已入职新公司三个月,rust技能从0开始,到现在已经可以熟练编写各种功能了. 这里分享一下笔者的rust学习路线. 在学习rust之前,笔者已经有三年多的C++开发经验,有一定的Python ...
- Rust个人学习之hello world
rust 作为新晋热门语言,作为软件研发工作者不得不开启新一轮的学习与思考,今天就考虑拿一本书作为入门,快速进入rust的世界. 既然是接触的一门新的编程语言,自然绕不过向这个美好的世界打声招呼,今天 ...
- rust军用船指令_Rust基础学习笔记(五):Cargo与Crates.io
最近复习鸽了几天,今天继续学,希望在考试之前搞定这个官方文档 もうダメ.... 本章学习Cargo和Crates.io相关,内容有下: 自定义构建 向crates.io提交libraries 利用工作 ...
- rust 入门笔记: rustlings(推荐一些学习rust语法的一些非常好的小练习)
rustlings 推荐一个学习rust非常好的repo: Small exercises to get you used to reading and writing Rust code! - 学习 ...
最新文章
- java异常体系结构详解
- C++ 类的静态成员(static)
- java统计文本中英文单词个数split_零基础入门NLP - 新闻文本分类Task2(天池入门赛)...
- 一个男的和计算机对话,父与子的对话:计算机算法
- jboss7.1.安装配置
- 人生:沉得住气,方成得了器!
- 如何HTML中输入正确格式,以HTML格式输入样式
- 对所谓原生软件的偏执
- 悬赏数据控!2018网易有数可视化大赛火热开启!
- 月薪一万的北漂可以过上什么样的生活?
- Java 爬取行政区划代码
- 计算机相关课程考核,计算机专业编程实践类课程考核方法
- 如何在职场上获得良好的起点
- 计算机网络验证性试验报告
- proteus7.7+Keil2仿真80C51控制流水灯
- AI人工智能可以做哪些课题的毕业设计毕设
- 微信专属输入法你有了吗?防窃听、护隐私
- Vi,Java,Ant,Junit的自学报告
- spring boot整合security时候处理静态资源拦截问题
- 视频编辑软件中滚动字幕如何能停下来
热门文章
- python setuptools 打包 发布
- java ADT的简单介绍
- android使用的数据,在 Android 应用中使用数据 - Xamarin | Microsoft Docs
- java需要了解的几个算法
- java list map 效率_遍历Map和List的几种方法和性能比较
- mysql MyISAM和InnoDB
- calendar与date区别及常用方法介绍
- 十六、Oracle学习笔记:索引和约束(表字段快速查询和约束)
- CentOS/Ubuntu 14.10 安装 phpstorm
- Oracle 11g r2 下载地址