我在几台具有低延迟内核的服务器上运行 Ubuntu 18。
以前我曾经chrt
提高与特定网络接口相关的中断的中断线程的优先级。
现在,在新安装的 Ubuntu 22 中,使用低延迟内核,我不再看到作为线程运行的网卡中断,因此我无法提高 chrt 的优先级。
有什么变化,有没有办法在使用 lowlatenct 内核的 Ubuntu 22 上增加网卡中断优先级?
我这样做的原因是我有几个应用程序实例,它以 10 毫秒的间隔与其他计算机同步少量数据,而这些数据一定不能延迟。
我确实找到了这有关 UbuntuStudio 的信息(虽然有点旧,是 2014 年的)线程中断内核参数,并且我已经确认我的新服务器上的内核是用 构建的CONFIG_IRQ_FORCED_THREADING=y
,但没有用 和CONFIG_IRQ_FORCED_THREADING_DEFAULT=y
而不是CONFIG_CMDLINE_BOOL=y
和CONFIG_CMDLINE="threadirqs"
。
它是否像使用线程中断范围如果是,那么低延迟内核何时以及为何发生变化,导致无法启动线程中断默认情况下?
细节
我的服务器先前的解决方案有效运行 Ubuntu 18.04.3 LTS,内核为 4.15.0-123-lowlatency
这些是我用来提高网络接口中断优先级的步骤:
- 我运行
grep eno1 /proc/interrupts
以获取网络接口的中断列表eno1
输出:
46: 19627 486510106 1600726600 752055110 1667993661 1758831968 1471240837 895245906 658188895 458440425 918559817 1494584384 1650824561 1492836078 1214143068 920349210 IR-PCI-MSI 5767169-edge i40e-eno1-TxRx-0
47: 782983498 3867361290 1350470347 1488832628 1644524179 1306935370 2202471485 885906069 266079923 769050734 767000710 1199122348 1933889985 2475859304 880359145 1207562098 IR-PCI-MSI 5767170-edge i40e-eno1-TxRx-1
48: 487247958 557149515 2436993704 1327150410 2327637329 1692350382 1160439225 1234994234 425076063 640500055 1555726813 1469038634 2133556615 1260984085 1076406628 1141316582 IR-PCI-MSI 5767171-edge i40e-eno1-TxRx-2
49: 1143028283 802247216 842065588 1507113921 1450442442 1285188073 1414989498 1386700444 961900607 949612962 1176201048 996374473 1392801596 1501055021 1244839988 1568153886 IR-PCI-MSI 5767172-edge i40e-eno1-TxRx-3
50: 668676162 585775185 627035622 560691459 545566733 534335154 675644369 619390835 602253370 611090233 610683285 552978104 558929823 827257425 555723069 751315724 IR-PCI-MSI 5767173-edge i40e-eno1-TxRx-4
51: 293791983 293018539 290310300 284911609 302624294 171231086 295128963 277943781 299153596 305718023 306559090 265176440 265877422 330072276 275946899 322208126 IR-PCI-MSI 5767174-edge i40e-eno1-TxRx-5
52: 191406211 164689451 184092472 163124351 197416416 147238980 150711931 166047824 174857188 176597379 163302040 163989899 169136092 206831749 154774311 195388230 IR-PCI-MSI 5767175-edge i40e-eno1-TxRx-6
53: 154062755 152443945 151052334 133925954 161462702 130698860 164068832 88730046 146570114 152839498 140082079 134023798 139525722 178502808 136857972 164157076 IR-PCI-MSI 5767176-edge i40e-eno1-TxRx-7
54: 689819152 457848749 1706899545 1350792167 1009016882 750940295 1280794513 1964042727 4293839311 604650430 1394985726 1390827662 1403478542 1032557167 1258596213 1421573353 IR-PCI-MSI 5767177-edge i40e-eno1-TxRx-8
55: 561179546 1569329752 1142403865 1952025068 1831162021 1786111309 1819812537 1726432246 809350013 3510981566 1252176327 1188689474 2184195381 1689746731 1808756473 1998075218 IR-PCI-MSI 5767178-edge i40e-eno1-TxRx-9
60: 922953009 535440685 1433001681 1370014373 1542680223 1564958146 1905615201 2245129577 606607580 751437652 653813572 1648944839 1339662143 2748851364 1328493925 2197584768 IR-PCI-MSI 5767179-edge i40e-eno1-TxRx-10
61: 915173255 898646194 818454435 975220367 1436058787 1258590833 1222669978 1475644158 955884853 1048337605 1028501060 1229229809 1375764724 1619404100 1174614194 1626915747 IR-PCI-MSI 5767180-edge i40e-eno1-TxRx-11
62: 663129689 560699700 659990638 564915959 716701151 537167244 709668792 653540652 643175469 609563694 657141242 551578294 496276894 826134713 584425798 782807812 IR-PCI-MSI 5767181-edge i40e-eno1-TxRx-12
63: 301178832 272290237 297349011 271058986 300930906 219283819 308344403 269722445 297793600 300198211 271567742 257075242 257586638 275439840 258617424 331559250 IR-PCI-MSI 5767182-edge i40e-eno1-TxRx-13
64: 161271542 160600034 162911736 135003940 159159336 126317895 145054883 142236108 153342852 158784173 149385736 130987700 145150304 179008100 100121849 168255307 IR-PCI-MSI 5767183-edge i40e-eno1-TxRx-14
65: 174086138 162484310 163432318 142315712 164872478 136966716 168251078 137840006 150045153 156520410 143270315 134481889 152034193 184135755 140295604 167224365 IR-PCI-MSI 5767184-edge i40e-eno1-TxRx-15
- 我运行
ps -el | grep irq
以查找每个 irq 线程的 pideno1
输出:(我仅显示与 irqs 46-65 匹配的相关行,还有更多)
1 R 0 1255 2 1 9 - - 0 - ? 6-01:09:56 irq/46-i40e-eno
1 S 0 1256 2 1 9 - - 0 - ? 5-13:32:39 irq/47-i40e-eno
1 S 0 1257 2 0 9 - - 0 - ? 5-04:09:18 irq/48-i40e-eno
1 S 0 1259 2 0 9 - - 0 - ? 3-12:50:35 irq/49-i40e-eno
1 S 0 1261 2 0 9 - - 0 - ? 1-19:03:44 irq/50-i40e-eno
1 S 0 1262 2 0 9 - - 0 - ? 19:55:05 irq/51-i40e-eno
1 S 0 1263 2 0 9 - - 0 - ? 11:48:06 irq/52-i40e-eno
1 S 0 1264 2 0 9 - - 0 - ? 09:59:36 irq/53-i40e-eno
1 S 0 1265 2 0 9 - - 0 - ? 5-09:23:00 irq/54-i40e-eno
1 S 0 1266 2 0 9 - - 0 - ? 4-19:20:49 irq/55-i40e-eno
1 S 0 1267 2 0 9 - - 0 - ? 4-19:24:25 irq/60-i40e-eno
1 S 0 1268 2 0 9 - - 0 - ? 3-10:21:49 irq/61-i40e-eno
1 S 0 1269 2 0 9 - - 0 - ? 1-20:29:13 irq/62-i40e-eno
1 S 0 1270 2 0 9 - - 0 - ? 19:24:51 irq/63-i40e-eno
1 S 0 1271 2 0 9 - - 0 - ? 10:11:11 irq/64-i40e-eno
1 S 0 1273 2 0 9 - - 0 - ? 10:31:14 irq/65-i40e-eno
- 对于此网络接口的每个 irq 线程,我运行
chrt -p 80 <thread-pid>
,例如chrt -p 80 1255
irq/46。
我实际上使用 grep、awk 和 xargs 的组合来一次性执行这三个步骤:
grep $NIC /proc/interrupts \
| awk '{print substr($1, 0, length($1) - 1);}' \
| xargs -I% sh -c "ps -e | grep irq/%-" \
| awk '{print $1;}' \
| xargs -I% sh -c "echo irq PID %; chrt -p 80 %"
me@myserver:~$ grep CONFIG_IRQ_FORCED_THREADING /boot/config-4.15.0-123-lowlatency
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_IRQ_FORCED_THREADING_DEFAULT=y
me@myserver:~$ grep CONFIG_CMDLINE /boot/config-4.15.0-123-lowlatency
CONFIG_CMDLINE_PARTITION=y
# CONFIG_CMDLINE_BOOL is not set
较新的服务器此解决方案不起作用的地方运行内核为 5.15.0-79-lowlatency 的 Ubuntu 22.04.2 LTS
前两步的结果:
grep eno4 /proc/interrupts
输出:
240: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1030184410 0 IR-PCI-MSI 5249025-edge i40e-eno4-TxRx-0
241: 0 0 0 0 0 0 0 0 0 0 0 0 1342657199 0 0 0 IR-PCI-MSI 5249026-edge i40e-eno4-TxRx-1
242: 0 0 1 1007985709 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249027-edge i40e-eno4-TxRx-2
243: 0 995307412 0 21 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249028-edge i40e-eno4-TxRx-3
244: 0 0 0 0 16 0 1231037185 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249029-edge i40e-eno4-TxRx-4
245: 0 0 0 0 782521466 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249030-edge i40e-eno4-TxRx-5
246: 1035173010 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249031-edge i40e-eno4-TxRx-6
247: 0 0 0 0 0 0 0 0 0 0 0 930082293 0 0 0 0 IR-PCI-MSI 5249032-edge i40e-eno4-TxRx-7
248: 0 0 0 0 0 0 0 762559502 1 0 0 0 0 0 0 0 IR-PCI-MSI 5249033-edge i40e-eno4-TxRx-8
249: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 793943748 0 IR-PCI-MSI 5249034-edge i40e-eno4-TxRx-9
250: 0 0 0 0 0 0 0 0 0 0 12 0 998551326 0 0 0 IR-PCI-MSI 5249035-edge i40e-eno4-TxRx-10
251: 0 0 0 0 0 0 0 0 0 0 975747642 1 0 0 0 0 IR-PCI-MSI 5249036-edge i40e-eno4-TxRx-11
252: 0 1342889081 0 0 0 0 0 0 0 0 0 0 216 0 0 0 IR-PCI-MSI 5249037-edge i40e-eno4-TxRx-12
253: 0 0 0 0 0 0 0 0 0 0 0 0 0 14 0 818489097 IR-PCI-MSI 5249038-edge i40e-eno4-TxRx-13
254: 0 0 0 0 1177974058 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 5249039-edge i40e-eno4-TxRx-14
255: 0 0 942945250 0 0 0 0 0 0 0 0 0 0 0 0 27 IR-PCI-MSI 5249040-edge i40e-eno4-TxRx-15
ps -el | grep irq
输出:
1 S 0 14 2 0 70 -10 - 0 - ? 00:06:33 ksoftirqd/0
1 S 0 23 2 0 70 -10 - 0 - ? 00:14:18 ksoftirqd/1
1 S 0 29 2 0 70 -10 - 0 - ? 00:30:44 ksoftirqd/2
1 S 0 35 2 0 70 -10 - 0 - ? 00:03:54 ksoftirqd/3
1 S 0 41 2 0 70 -10 - 0 - ? 00:09:49 ksoftirqd/4
1 S 0 47 2 0 70 -10 - 0 - ? 00:02:08 ksoftirqd/5
1 S 0 53 2 0 70 -10 - 0 - ? 00:06:32 ksoftirqd/6
1 S 0 59 2 0 70 -10 - 0 - ? 00:05:31 ksoftirqd/7
1 S 0 65 2 0 70 -10 - 0 - ? 00:03:34 ksoftirqd/8
1 S 0 71 2 0 70 -10 - 0 - ? 00:00:29 ksoftirqd/9
1 S 0 77 2 0 70 -10 - 0 - ? 00:04:11 ksoftirqd/10
1 S 0 83 2 0 70 -10 - 0 - ? 00:02:47 ksoftirqd/11
1 S 0 89 2 0 70 -10 - 0 - ? 00:13:01 ksoftirqd/12
1 S 0 95 2 0 70 -10 - 0 - ? 00:01:02 ksoftirqd/13
1 S 0 101 2 0 70 -10 - 0 - ? 00:06:02 ksoftirqd/14
1 S 0 107 2 0 70 -10 - 0 - ? 00:06:08 ksoftirqd/15
1 I 0 197 2 0 60 -20 - 0 - ? 00:00:00 vfio-irqfd-clea
4 S 0 924 1 0 80 0 - 20764 - ? 00:10:46 irqbalance
内核构建参数:
me@myserver:~$ grep CONFIG_IRQ_FORCED_THREADING /boot/config-5.15.0-79-lowlatency
CONFIG_IRQ_FORCED_THREADING=y
me@myserver:~$ grep CONFIG_CMDLINE_ /boot/config-5.15.0-79-lowlatency
# CONFIG_CMDLINE_BOOL is not set
CONFIG_CMDLINE_PARTITION=y