RHEL8 systemd 使用大量 CPU,不执行任何操作时系统负载非常高

RHEL8 systemd 使用大量 CPU,不执行任何操作时系统负载非常高

我一直在尝试确定系统高负载的来源,

顶部:

top - 09:55:12 up 1 day, 11:48,  4 users,  load average: 7.64, 6.52, 6.33
Tasks: 279 total,   2 running, 276 sleeping,   0 stopped,   1 zombie
%Cpu(s): 50.0 us, 50.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1785.3 total,    223.3 free,    938.8 used,    623.2 buff/cache
MiB Swap:   4096.0 total,   2259.5 free,   1836.5 used.    664.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                
1267200 root      20   0    0.1g   0.0g   0.0g R  50.0   0.3   0:00.61 top                                                                    
      1 root      20   0    0.2g   0.0g   0.0g S   0.0   0.3 347:46.38 /usr/lib/systemd/systemd --switched-root --system --deserialize 17     
      2 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.15 [kthreadd]                                                             
      3 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [rcu_gp]                                                               
      4 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [rcu_par_gp]                                                           
      6 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [kworker/0:0H-events_highpri]                                          
      9 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [mm_percpu_wq]                                                         
     10 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [rcu_tasks_rude_]                                                      
     11 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [rcu_tasks_trace]                                                      
     12 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   6:10.89 [ksoftirqd/0]                                                          
     13 root      20   0    0.0g   0.0g   0.0g I   0.0   0.0   0:41.25 [rcu_sched]                                                            
     14 root      rt   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [migration/0]                                                          
     15 root      rt   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.12 [watchdog/0]                                                           
     16 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [cpuhp/0]                                                              
     18 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [kdevtmpfs]                                                            
     19 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [netns]                                                                
     20 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:03.12 [kauditd]                                                              
     21 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.11 [khungtaskd]                                                           
     22 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [oom_reaper]                                                           
     23 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.01 [writeback]                                                            
     24 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   0:01.01 [kcompactd0]                                                           
     25 root      25   5    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [ksmd]                                                                 
     26 root      39  19    0.0g   0.0g   0.0g S   0.0   0.0   0:07.19 [khugepaged]                                                           
     27 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [crypto]                                                               
     28 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [kintegrityd]                                                          
     29 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [kblockd]                                                              
     30 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [blkcg_punt_bio]                                                       
     31 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [tpm_dev_wq]                                                           
     32 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [md]                                                                   
     33 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [edac-poller]                                                          
     34 root      rt   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [watchdogd]                                                            
     35 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:10.06 [kworker/0:1H-kblockd]                                                 
     69 root      20   0    0.0g   0.0g   0.0g S   0.0   0.0   5:12.96 [kswapd0]                                                              
    171 root       0 -20    0.0g   0.0g   0.0g I   0.0   0.0   0:00.00 [kthrotld]                                                             
    172 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/24-pciehp]                                                        
    173 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/25-pciehp]                                                        
    174 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/26-pciehp]                                                        
    175 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/27-pciehp]                                                        
    176 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/28-pciehp]                                                        
    177 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/29-pciehp]                                                        
    178 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/30-pciehp]                                                        
    179 root     -51   0    0.0g   0.0g   0.0g S   0.0   0.0   0:00.00 [irq/31-pciehp]       

虚拟机统计:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 6  0 1676376  66624 929468 403424  109  160  1073   274  120   95 13 18 69  0  0

iostat:

Linux 4.18.0-425.13.1.el8_7.x86_64 (hostname)   04/20/2023  _x86_64_    (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          12.30    0.82   17.65    0.26    0.00   68.97

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              38.05      1064.79       272.12  138382106   35365399
dm-0             10.39       250.39        25.20   32540656    3274422
dm-1             66.44       107.65       158.12   13989760   20548928
dm-2              2.39       147.00        11.90   19103775    1546792
dm-3              0.00         0.03         0.02       3989       2082
dm-4              3.45       152.32        31.33   19795688    4071283
dm-5              0.30         2.01         5.01     260581     651253
dm-6              0.00         0.03         0.02       3440       2048
dm-7              0.02         0.06         0.18       8149      23581
dm-8              0.06         0.80         0.27     104186      34788
dm-9              0.30         9.03         0.11    1173075      14455
dm-10             8.83       387.11        39.96   50309334    5192814
loop0             0.00         0.01         0.00       1917        101
loop1             0.00         0.03         0.00       3332          1

如何进一步调试导致此问题的原因?

相关内容