☰

Proxmox PCIe 直通无法工作

tag-icon • tag-icon qemu proxmox
Proxmox PCIe 直通无法工作

我正在尝试通过网络设备传递到运行 pfSense 的虚拟机，但无法使其正常工作。当我启动 pfSense 时，网卡根本不可见。我认为我已经正确设置了所有内容，有人可以提出问题是什么吗？

安装的 PCI 设备有：

# lspci -nn
<snip>
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
01:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
02:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
03:00.0 Network controller [0280]: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] [8086:24fb] (rev 10)

我要通过的设备是01:00.0

列出硬件返回：

# Ishw -class network
<snip>
*-network
description: Ethernet controller
product: 82574L Gigabit Network Connection
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: pm msi pciexpress msix cap_list rom
configuration: driver=vfio-pci latency=0
resources: irq:16 memory:a13c0000-a13dffff memory:a1300000-a137ffff ioport:4000(size=32) memory:a13e0000-a13e3fff memory:a1380000-a13bffff

VMID 为 100，/etc/pve/qemu-server/100.conf 包含：

bootdisk: virtio0
cores: 2
hostpci1: 01:00,pcie=1,rombar=0
ide2: NAS1:iso/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso,media=cdrom
machine: q35
memory: 2048
name: pfSense
net0: virtio=16:B9:C8:87:DE:54,bridge=vmbr1,firewall=1
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=a18cecad-2bf6-4e81-ad32-aa2b2d7d0142
sockets: 1
usb0: host=148f:2573,usb3=1
virtio0: local-lvm:vm-100-disk-0,size=32G
vmgenid: 412dae1f-41c4-49eb-9a58-398bd33c350b

/etc/default/grub 有以下内容：

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

/etc/modules 包含：

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

验证它是否全部启用（这与 doco 有点不同）

# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.012056] ACPI: DMAR 0x000000007E41DBC0 0000A8 (v01 INTEL EDK2 00000002 01000013)
[ 0.182375] DMAR: IOMMU enabled
[ 0.349186] DMAR: Host address width 39
[ 0.349187] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.349192] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.349193] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.349196] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.349197] DMAR: RMRR base: 0x0000007e704000 end: 0x0000007e94dfff
[ 0.349198] DMAR: RMRR base: 0x0000007f800000 end: 0x0000008fffffff
[ 0.349200] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.349201] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.349201] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.351514] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.315529] DMAR: No ATSR found
[ 1.315585] DMAR: dmar0: Using Queued invalidation
[ 1.315588] DMAR: dmar1: Using Queued invalidation
[ 1.345917] DMAR: Intel(R) Virtualization Technology for Directed I/O

检查它们都在不同的组中返回：

# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/3/devices/0000:00:12.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.6
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:02:00.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.5
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3
/sys/kernel/iommu_groups/9/devices/0000:00:1f.6
/sys/kernel/iommu_groups/9/devices/0000:00:1f.4

最后，检查所有设置是否正确：

# lspci -nnk
<snip>
01:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
Subsystem: Intel Corporation Gigabit CT Desktop Adapter [8086:a01f]
Kernel driver in use: vfio-pci
Kernel modules: e1000e

根据文档，最后一步表明应该一切顺利......所以我不确定发生了什么。

答案1

我发现了这个问题。 pfSense 使用 freeBSD，并且 freeBSD 中存在一个阻止 PCIe 直通的错误。 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243640

我通过创建 Fedora 31 VM 并成功通过 WiFi PCIe NIC 来验证这是我的问题。

解决方案是等待 freeBSD 修补内核并等待 pfSense 合并它。我为 pfSense 提出了一个错误，以便他们可以跟踪它（https://redmine.pfsense.org/issues/10315）

关于使用 i440fx 的解决方法有很多讨论，但我无法让它发挥作用。

如果有人能够就如何实施解决方法提供一些指导，那就太好了。

相关内容