我在 Alienware Aurora R6（i7-7700 CPU 和 Nvidia GTX 1080）上安装了 Ubuntu Server 20.04 LTS。我对类型 1 虚拟机管理程序比较陌生，对类型 2（VirtualBox）有一些经验。
在主机系统上，我使用 Cockpit 创建了一个 Xubuntu 20.04 QEMU VM 作为测试实验室。我将其配置为
- 支持CPU直通
- 支持 GTX 1080 的 PCI 直通，还包含针对特定 GPU 的自定义 VBIOS（在此之前使用NVIDIA vBIOS VFIO 修补程序
- 连接到桥接网络接口，将其绑定到主机的以太网连接，以实现远程（SSH、VNC）、内部（研究所网络）和 Internet 访问
以下是各个设备的配置（完整配置可在文章末尾看到）：
中央处理器
<cpu mode='host-passthrough' check='partial'>
<topology sockets='1' cores='6' threads='1'/>
<cache mode='passthrough'/>
</cpu>
网络
<interface type='bridge'>
<mac address='52:54:00:2a:b8:4f'/>
<source bridge='bridge_vm_eth'/>
<model type='virtio'/>
<link state='up'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</interface>
图形处理器
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<rom file='/home/gin/gpu_roms/Dell.GTX1080.8192.170320.rom'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
</hostdev>
棘手的部分是 GPU，因为我以前从来没有做过这样的事情，而且基于此教程我为 VGA 和声音控制器创建了一个直通，它们都是 Nvidia GPU 的一部分，并且是工作驱动程序安装所必需的。
我曾经lspci -nn | grep -i nvidia获取过 PCI BDF（总线、设备、功能）以及供应商和设备 ID（使用此教程关于输出中哪一列是什么的解释):
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
并用它来为我的配置中的两个条目分配源地址<hostdev/>。客户 PCI 插槽的分配没有经过太多考虑（只是查看其余部分并查看下一个可用的总线/插槽号）。我也在multifunction图形上激活了<hostdev/>，但不确定是否也需要对声音这样做。
我的 GRUB 配置如下：
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=0
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="maybe-ubiquity intel_iommu=on vfio-pci.ids=10de:1b80,10de:10f0"
GRUB_CMDLINE_LINUX
我在其中启用了英特尔的 IOMMU，并将 PCI ID（供应商：设备）添加到 PCI ID 的 VFIO 列表中。
此外我的/etc/modprobe.d/vfio.conf只包含
options vfio-pci ids=10de:1b80,10de:10f0
为了能够安装主机操作系统，我还必须做两件事：禁用安全启动并打开传统模式（也许这也是我下面问题的根源），以及none由于 AppArmor 的问题而将安全驱动程序设置为（目前正在调查修复方法）。
在安装 Xubuntu 20.04 作为我的客户虚拟机后，我做了一些配置（代理、X11VNC、SSH 等），并立即通过运行命令检查是否检测到 GPU lspci（参见上文）：
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
06:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
如您所见，操作系统将 VGA 和声音控制器都检测为 PCI 设备，并且分配的 PCI 地址（参见<hostdev/>上面的 XML 片段）以及供应商和设备 ID 匹配。
我继续安装推荐的 Nvidia 驱动程序
sudo ubuntu-drivers autoinstall
这使得 Nvidia 驱动程序 470（经过测试，推荐）安装成功（安装过程中没有错误消息）。我还尝试使用驱动程序版本 390 和 510 来诊断下面的问题，这两个版本被列为 Ubuntu 18.04 的附加驱动程序。
目前我的虚拟机正在使用 QXL，根据我的虚拟机的配置，它也是主要的视频输出设备：
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
并且 - 至少据我所知 - 是一种图形控制器的基本模拟，其工作方式（至少表面上）与集成显卡非常相似，其中 RAM 用作 VRAM，CPU 提供渲染功能。
这一切都很好，因为我想将 Nvidia GPU 用于其他用途，即 CUDA（当然后来我得到了一个令人遗憾的事实，CUDA 8.0 工具包，这是该工具包的最后一个支持版本，在 Ubuntu 16.04 上得到官方支持，在 18.04 上有一些成功案例，但在 20.04 上没有）所以卸载专用 GPU 是完全没问题的。
为了测试，我安装了 DaVinci Resolve 17。启动时，我收到一条消息，称不支持 GPU 处理模式，需要进一步配置。不用说，我得到了
显然，可用 GPU 列表为空
接下来我要做的是加载 Nvidia X 服务器设置（首先是 GUI，然后是终端）。我收到了
从终端启动后，我得到了
ERROR: An internal driver error occurred
ERROR: Unable to load info from any available system
(nvidia-settings:1536): GLib-GObject-CRITICAL **: 11:02:41.340: g_object_unref: assertion 'G_IS_OBJECT (object)' failed
** Message: 11:02:41.343: PRIME: No offloading required. Abort
** Message: 11:02:41.343: PRIME: is it supported? no
当然进一步nvidia-smi给出
Unable to determine the device handle for GPU 0000:06:00.0: Unknown Error
不用说，这是一条非常具有描述性的错误消息。
知道我遗漏了什么吗？我读过有关iommu=pt提高性能或臭名昭著的错误 43 的文章，根据我的研究，这至少不应该适用于 Linux 上的最新驱动程序。
更新
以下是来自虚拟机我觉得有趣的是：
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.485748] nvidia: module license 'NVIDIA' taints kernel.
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.485753] Disabling lock debugging due to kernel taint
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.488862] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.5/0000:06:00.1/sound/card1/input5
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.488890] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.5/0000:06:00.1/sound/card1/input6
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.488911] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.5/0000:06:00.1/sound/card1/input7
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.488933] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.5/0000:06:00.1/sound/card1/input8
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.488956] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:02.5/0000:06:00.1/sound/card1/input9
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.488979] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:02.5/0000:06:00.1/sound/card1/input10
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.489001] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:02.5/0000:06:00.1/sound/card1/input11
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.508386] nvidia: module verification failed: signature and/or required key missing - tainting kernel
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.530278] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.530287]
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.537228] nvidia 0000:06:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.739718] cryptd: max_cpu_qlen set to 1000
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.743814] AVX2 version of gcm_enc/dec engaged.
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 4.743828] AES CTR mode by8 optimization enabled
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 5.015914] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 470.103.01 Thu Jan 6 12:10:04 UTC 2022
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 5.025646] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 470.103.01 Thu Jan 6 >
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 5.027257] [drm] [nvidia-drm] [GPU ID 0x00000600] Loading driver
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 5.027259] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:06:00.0 on minor 1
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 5.036862] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
Mar 30 09:33:04 SZA-DT043-L-VM0 kernel: [ 5.038962] nvidia-uvm: Loaded the UVM driver, major device number 509.
...
Mar 30 09:33:11 SZA-DT043-L-VM0 kernel: [ 13.548312] NVRM: GPU 0000:06:00.0: RmInitAdapter failed! (0x24:0xffff:1211)
Mar 30 09:33:11 SZA-DT043-L-VM0 kernel: [ 13.548440] NVRM: GPU 0000:06:00.0: rm_init_adapter failed, device minor number 0
Mar 30 09:33:11 SZA-DT043-L-VM0 kernel: [ 13.715984] NVRM: GPU 0000:06:00.0: RmInitAdapter failed! (0x24:0xffff:1211)
Mar 30 09:33:11 SZA-DT043-L-VM0 kernel: [ 13.716118] NVRM: GPU 0000:06:00.0: rm_init_adapter failed, device minor number 0
Mar 30 09:33:18 SZA-DT043-L-VM0 kernel: [ 21.252309] input: spice vdagent tablet as /devices/virtual/input/input12
Mar 30 09:33:19 SZA-DT043-L-VM0 kernel: [ 21.940404] NVRM: GPU 0000:06:00.0: RmInitAdapter failed! (0x24:0xffff:1211)
Mar 30 09:33:19 SZA-DT043-L-VM0 kernel: [ 21.940534] NVRM: GPU 0000:06:00.0: rm_init_adapter failed, device minor number 0
Mar 30 09:33:20 SZA-DT043-L-VM0 kernel: [ 22.332615] NVRM: GPU 0000:06:00.0: RmInitAdapter failed! (0x24:0xffff:1211)
Mar 30 09:33:20 SZA-DT043-L-VM0 kernel: [ 22.332750] NVRM: GPU 0000:06:00.0: rm_init_adapter failed, device minor number 0
内核日志来自主持人包含
...
Mar 30 10:37:54 SZA-DT043-L kernel: [416243.385098] snd_hda_intel 0000:01:00.1: Disabling MSI
Mar 30 10:37:54 SZA-DT043-L kernel: [416243.385117] snd_hda_intel 0000:01:00.1: Handle vga_switcheroo audio client
Mar 30 10:37:56 SZA-DT043-L kernel: [416245.129722] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input28
Mar 30 10:37:56 SZA-DT043-L kernel: [416245.129916] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input29
Mar 30 10:37:56 SZA-DT043-L kernel: [416245.130094] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input30
Mar 30 10:37:56 SZA-DT043-L kernel: [416245.130283] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input31
Mar 30 10:37:56 SZA-DT043-L kernel: [416245.130432] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input32
Mar 30 10:37:56 SZA-DT043-L kernel: [416245.130582] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input33
Mar 30 10:37:56 SZA-DT043-L kernel: [416245.130729] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card1/input34
...
Mar 30 11:10:54 SZA-DT043-L kernel: [ 134.197857] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xd0000000-0xd1ffffff 64bit pref]
...
Mar 30 11:11:03 SZA-DT043-L kernel: [ 143.752631] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xd0000000-0xd1ffffff 64bit pref]
VM 完整配置 XML 文件
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit Xubuntu20.04
or other application using the libvirt API.
-->
<domain type='kvm'>
<name>Xubuntu20.04</name>
<uuid>3534d4f4-c899-402c-b82b-ea34b4b9b65e</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://ubuntu.com/ubuntu/20.04"/>
</libosinfo:libosinfo>
<cockpit_machines:data xmlns:cockpit_machines="https://github.com/cockpit-project/cockpit/tree/master/pkg/machines">
<cockpit_machines:has_install_phase>false</cockpit_machines:has_install_phase>
<cockpit_machines:install_source_type>disk_image</cockpit_machines:install_source_type>
<cockpit_machines:install_source>/etc/libvirt/qemu/Xubuntu20.04.xml</cockpit_machines:install_source>
<cockpit_machines:os_variant>ubuntu20.04</cockpit_machines:os_variant>
</cockpit_machines:data>
</metadata>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<vcpu placement='static' current='1'>6</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<vmport state='off'/>
</features>
<cpu mode='host-passthrough' check='partial'>
<topology sockets='1' cores='6' threads='1'/>
<cache mode='passthrough'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='volume' device='disk'>
<driver name='qemu' type='qcow2'/>
<source pool='default' volume='Xubuntu20.04.qcow2'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x2'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='7' port='0x16'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:1e:cf:8b'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:2a:b8:4f'/>
<source bridge='bridge_vm_eth'/>
<model type='virtio'/>
<link state='up'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
<image compression='off'/>
</graphics>
<graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
<listen type='address' address='127.0.0.1'/>
</graphics>
<sound model='ich9'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<rom file='/home/gin/gpu_roms/Dell.GTX1080.8192.170320_PATCHED.rom'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
</hostdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</rng>
</devices>
</domain>
答案1
我按照 reddit 上的建议删除了自定义 ROM。这没有帮助，但至少消除了当前问题的一个可能来源。
基于本文我检查了我的 Nvidia 所属的 IOMMU 组，发现它还包含一个英特尔 PCI 桥：
IOMMU Group 1:
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
移除 PCI 桥后
echo 1 > /sys/bus/pci/devices/0000\:00\:03.1/remove
echo 1 > /sys/bus/pci/rescan
重启虚拟机后，我就能修复这个问题了。为了使修复永久生效，我在准备阶段包含上述代码并将其存储在里面，/etc/libvirt/hooks/qemu.d/Xubuntu20.04/prepare/begin/script.sh以便libvirt自动加载它。