如何在 ubuntu 22.04 LTS 上为 NIC(或 SR-IOV)启用 PCI 直通

如何在 ubuntu 22.04 LTS 上为 NIC(或 SR-IOV)启用 PCI 直通

我有一台 DL380p G9,带有 4 端口板载 NIC 适配器。虽然制造商说它支持 SR-IOV。但我在 OS 命令输出中没有看到此功能。我打算为虚拟机使用 3 个端口,为主机使用 1 个端口。

如何让该适配器支持 PCI PAssthrough(或 SR-IOV)?

谢谢。

网卡:

# lspci -nn|grep Ether
02:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
02:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
02:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)
02:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01)

供应商: 位于:https://www.broadcom.com/products/ethernet-connectivity/network-adapters/bcm5719-1gbase-t-ic

主机接口支持每个 LAN 接口的单独 PCI Express 功能。BCM5719 包括 I/O 虚拟化 (IOV) 功能,例如 17 个接收队列和 16 个传输队列,以及具有灵活向量到队列关联的 17 个 MSI-X 向量。这些 IOV 功能使 BCM5719 能够支持 VMWare® NetQueue 和 Microsoft® VMQ 技术。

每 HP: https://support.hpe.com/hpesc/public/docDisplay?docId=c04249126&docLocale=en_US

对于支持单根 I/O 虚拟化 (SR-IOV) 的 HP Broadcom NIC,可以在虚拟化主机上创建虚拟功能并将其分配给虚拟化客户机。

在操作系统中:

# lspci -vvv -s 02:00.0
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
        DeviceName: Embedded LOM 1 Port 1
        Subsystem: Hewlett-Packard Company Ethernet 1Gb 4-port 331i Adapter
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        NUMA node: 0
        Region 0: Memory at 92b90000 (64-bit, prefetchable) [size=64K]
        Region 2: Memory at 92ba0000 (64-bit, prefetchable) [size=64K]
        Region 4: Memory at 92bb0000 (64-bit, prefetchable) [size=64K]
        Expansion ROM at 93000000 [virtual] [disabled] [size=256K]
        Capabilities: [48] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] Vital Product Data
                Product Name: HPE Ethernet 1Gb 4-port 331i Adapter
                Read-only fields:
                        [PN] Part number: N/A
                        [EC] Engineering changes: A-5427
                        [SN] Serial number: N/A
                        [MN] Manufacture ID: 103C
                        [V0] Vendor specific: 5W PCIeGen2
                        [V1] Vendor specific: 21.6.28
                        [V2] Vendor specific: N/A
                        [V3] Vendor specific: 1.46
                        [V5] Vendor specific: 0A
                        [V6] Vendor specific: 21.6.0
                        [V7] Vendor specific: 331i
                        [VA] Vendor specific: 20.19.51
                        [RV] Reserved: checksum good, 215 byte(s) reserved
                Read/write fields:
                        [RW] Read-write area: 10 byte(s) free
                End
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [a0] MSI-X: Enable+ Count=17 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00001000
        Capabilities: [ac] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr+ NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM not supported
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x2 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [13c v1] Device Serial Number 00-00-ec-b1-d7-7f-9a-90
        Capabilities: [150 v1] Power Budgeting <?>
        Capabilities: [160 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
        Kernel driver in use: tg3
        Kernel modules: tg3

我看不到这些功能。是驱动程序问题吗?

++添加 BIOS 截图:

BIOS 设置 - 1

BIOS 网络设置 - 2

BIOS 设置 - 3

答案1

答案是:这不可能。宣传的 SR-IOV 功能仅适用于 VMWare 和 Windows。不支持 Linux :-(

来自文档:BCM5719 文档

VMWare NetQueue 和 Microsoft VMQ 的 I/O 虚拟化支持

– 每个端口有 17 个接收队列和 16 个传输队列

– 17 个 MSI-X 向量支持每个队列中断到主机

答案2

就我而言,我必须在系统实用程序中启用 SR-IOV 功能,并在 BIOS 内部设置每个卡的最大 VF,然后才能在内核上看到 num_vfs。

BIOS 设置:

BIOS 设置

相关内容