Unix SORT 不尊重其内存限制

Unix SORT 不尊重其内存限制

我面临着由于过度使用内存而被杀死的问题sort,并且无法理解为什么它大大超出了给定的限制。

Context: 
Available physical RAM: 8GB 
Memory in use by system: <500MB (FreeBSD 13) 
Free memory: ~7.5GB 
Swap: 2G
No system hard limit on mem usage ($ ulimit -> unlimited)
File being sorted: 300+ GB 
HDD space: 1Tb+ free both on target or temporary dir.

即使当我运行排序时指定可用资源限制;或者没有任何明确的参数(应自动设置为可用的 90%),它最终会因过度使用而被杀死。

[mario@casa-server ~/pjy]$ sort -S 7G -T /filesWD-WMAY03648941/tmp/ -o /filesWD-WMAY03648941/pjA-sorted.txt pjyA.txt

Killed
[mario@casa-server ~/pjy]$
[mario@casa-server ~/pjy]$ sort -T /filesWD-WMAY03648941/tmp/ -o /filesWD-WMAY03648941/pjA-sorted.txt pjyA.txt

Killed
[mario@casa-server ~/pjy]$

显然它确实超出了资源限制。当使用 -S 3G 运行时,我有时可以观察到大约 5G 的使用量。

# ~5G USAGE EVEN WITH EXPLICIT -S 3G LIMIT


  [mario@casa-server ~]$ sudo procstat -r 3183
  PID COMM             RESOURCE                          VALUE        
 3183 sort             user time                    00:27:51.849171   
 3183 sort             system time                  00:03:28.220262   
 3183 sort             maximum RSS                          5423072 KB
 3183 sort             integral shared memory              10505728 KB
 3183 sort             integral unshared data               1910064 KB
 3183 sort             integral unshared stack             30560128 KB
 3183 sort             page reclaims                        8172561   
 3183 sort             page faults                             1724   
 3183 sort             swaps                                      0   
 3183 sort             block reads                            24296   
 3183 sort             block writes                           17719   
 3183 sort             messages sent                              0   
 3183 sort             messages received                          0   
 3183 sort             signals received                           0   
 3183 sort             voluntary context switches              5868   
 3183 sort             involuntary context switches           30700   
[mario@casa-server ~]$ 


# TOP excerpt
 PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 3183 mario         1 103    0  4657M  4603M CPU3     3  38:23  99.88% sort -S 3G -T /filesWD-WMAY03648941/tmp/ -o /files



# NO SO LIMIT ON USAGE
[mario@casa-server ~]$ sudo procstat -l 3183
  PID COMM             RLIMIT                  SOFT             HARD     
 3183 sort             cputime             infinity         infinity     
 3183 sort             filesize            infinity         infinity     
 3183 sort             datasize               32768 MB         32768 MB  
 3183 sort             stacksize             524288 KB        524288 KB  
 3183 sort             coredumpsize        infinity         infinity     
 3183 sort             memoryuse           infinity         infinity     
 3183 sort             memorylocked           65536 B          65536 B   
 3183 sort             maxprocesses           12186            12186     
 3183 sort             openfiles             234576           234576     
 3183 sort             sbsize              infinity         infinity     
 3183 sort             vmemoryuse          infinity         infinity     
 3183 sort             pseudo-terminals    infinity         infinity     
 3183 sort             swapuse             infinity         infinity     
 3183 sort             kqueues             infinity         infinity     
 3183 sort             umtxp               infinity         infinity     
[mario@casa-server ~]$
[mario@casa-server ~]$ ulimit 
unlimited

这就解释了为什么它在以 7G 限制(7.5 可用)运行时被杀死,或者当限制被自动设置时:它确实超过了使用所有 mem+swap(并按预期被操作系统杀死)。

但我在这里缺少什么?man sort:

 -S size, --buffer-size=size
         Use size for the maximum size of the memory buffer.  Size
         modifiers %,b,K,M,G,T,P,E,Z,Y can be used.  If a memory limit is
         not explicitly specified, sort takes up to about 90% of available
         memory.  If the file size is too big to fit into the memory
         buffer, the temporary disk files are used to perform the sorting.

如果-S 3G设置了,为什么会超过使用RAM而不是HDD临时文件夹?

PS:我试图问这个与其他“由于内存不足而被杀”相关的问题,但一些版主管理不善。我确实认为这是一个相关的问题(将其标记为不相关的问题只会使解决方案支离破碎)。另外,我阅读了许多其他可能的解决方案,但它们不适合这种情况。

谢谢大家

相关内容