参数列表和导出变量的总大小超出操作系统 1MB 的限制

参数列表和导出变量的总大小超出操作系统 1MB 的限制

我正在对一些科学计算项目进行网格搜索,并得到了一堆文件名中嵌入了参数的日志文件。

$ ls -l | wc -l
   38210
$ ls -l | tail
-rw-rw-r--  1 qys  wheel  1070 Jul 20 15:01 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.25.txt
-rw-rw-r--  1 qys  wheel  1070 Jul 20 15:00 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.5.txt
-rw-rw-r--  1 qys  wheel  1070 Jul 20 15:01 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.75.txt
-rw-rw-r--  1 qys  wheel  1070 Jul 20 15:00 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa0.txt
-rw-rw-r--  1 qys  wheel  1070 Jul 20 15:01 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda0_kappa1.txt
-rw-rw-r--  1 qys  wheel   981 Jul 20 15:06 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.25.txt
-rw-rw-r--  1 qys  wheel   981 Jul 20 15:08 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.5.txt
-rw-rw-r--  1 qys  wheel   981 Jul 20 15:08 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.75.txt
-rw-rw-r--  1 qys  wheel   981 Jul 20 15:06 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa0.txt
-rw-rw-r--  1 qys  wheel   984 Jul 20 15:08 gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa1.txt

每个文件看起来都像

$ cat gmm_dim96_aelr0.01_ufactor100_K8_R8_lr1_lambda1_kappa1.txt
GMM: epoch 0: train loss 662699.9375, valid accuracy 80.03%
GMM: epoch 1: train loss 662553.75, valid accuracy 79.92%
GMM: epoch 2: train loss 662446.1875, valid accuracy 79.77%
GMM: epoch 3: train loss 662429.5, valid accuracy 79.82%
GMM: epoch 4: train loss 662404.5625, valid accuracy 79.83%
GMM: epoch 5: train loss 662395.0625, valid accuracy 79.85%
GMM: epoch 6: train loss 662381.25, valid accuracy 79.85%
GMM: epoch 7: train loss 662370.25, valid accuracy 79.83%
GMM: epoch 8: train loss 662362.875, valid accuracy 79.86%
GMM: epoch 9: train loss 662353.6875, valid accuracy 79.84%
GMM: epoch 10: train loss 662343.4375, valid accuracy 79.83%
GMM: epoch 11: train loss 662335.0625, valid accuracy 79.83%
GMM: epoch 12: train loss 662325.5, valid accuracy 79.85%
GMM: epoch 13: train loss 662316.6875, valid accuracy 79.87%
GMM: epoch 14: train loss 662307.0625, valid accuracy 79.88%
GMM: epoch 15: train loss 662298.0625, valid accuracy 79.90%
GMM: end: test accuracy 80.38%

我想通过评估精度对参数配置进行排序grep,但出现以下错误

$ grep 'GMM: epoch 15: ' gmm_*ufactor100*.txt | sort -k 9 -n -r | head
exec: Failed to execute process '/usr/bin/grep': the total size of the argument list and exported variables (2.1MB) exceeds the OS limit of 1MB.

如何允许更大尺寸的参数列表?我使用的是 macOS Monterey,以防相关。

相关内容