GNU 可以在程序退出之前并行输出 stdout 吗?

GNU 可以在程序退出之前并行输出 stdout 吗?
echo 'echo "hello, world!";sleep 3;' | parallel

此命令在完成之前不会输出任何内容。 Parallel 的手册页声称:

GNU 并行确保命令的输出与顺序运行命令时得到的输出相同。

我猜魔鬼就在措辞中:你会得到与正常运行它相同的输出,但不会得到与正常运行它相同的输出。例如,我一直在寻找可以执行此操作的选项--results /dev/stdout,但这不起作用。

我的用例是查看我正在运行的命令的实时进度输出。这不是关于已完成多少任务、可以为我显示哪些并行任务,而是关于我想查看的每个命令的进度输出。

我会使用 bash 循环 ( for i in $x; do cmd & done;),但我希望能够使用单个 Ctrl+C 停止所有任务,并行允许我这样做。

是否可以并行执行此操作?如果没有,还有其他工具吗?

答案1

我想你正在寻找--ungroup。手册页说:

--group  Group output. Output from each jobs is grouped 
         together and is only printed when the command is finished. 

         --group is the default. Can be reversed with -u.

-u当然是 的同义词--ungroup

答案2

要查看一些并行作业的进度,请尝试--tmuxpane --fg

parallel --tmuxpane --fg seq {} 10000000 ::: {1..100}

您也可能正在寻找-u或 (更有可能)--lb。从man parallel

   --line-buffer
   --lb
       Buffer output on line basis. --group will keep the output together
       for a whole job. --ungroup allows output to mixup with half a line
       coming from one job and half a line coming from another job.
       --line-buffer fits between these two: GNU parallel will print a full
       line, but will allow for mixing lines of different jobs.

       --line-buffer takes more CPU power than both --group and --ungroup,
       but can be much faster than --group if the CPU is not the limiting
       factor.

       Normally --line-buffer does not buffer on disk, and can thus process
       an infinite amount of data, but it will buffer on disk when combined
       with: --keep-order, --results, --compress, and --files. This will
       make it as slow as --group and will limit output to the available
       disk space.

       With --keep-order --line-buffer will output lines from the first job
       while it is running, then lines from the second job while that is
       running. It will buffer full lines, but jobs will not mix. Compare:

         parallel -j0 'echo {};sleep {};echo {}' ::: 1 3 2 4
         parallel -j0 --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4
         parallel -j0 -k --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4

       See also: --group --ungroup

[...]

   --ungroup
   -u  Ungroup output.  Output is printed as soon as possible and by passes
       GNU parallel internal processing. This may cause output from
       different commands to be mixed thus should only be used if you do not
       care about the output. Compare these:

         seq 4 | parallel -j0 \
           'sleep {};echo -n start{};sleep {};echo {}end'
         seq 4 | parallel -u -j0 \
           'sleep {};echo -n start{};sleep {};echo {}end'

       It also disables --tag. GNU parallel outputs faster with -u. Compare
       the speeds of these:

         parallel seq ::: 300000000 >/dev/null
         parallel -u seq ::: 300000000 >/dev/null
         parallel --line-buffer seq ::: 300000000 >/dev/null

       Can be reversed with --group.

       See also: --line-buffer --group

一个闪亮的例子-u是 stdout 和 stderr 混合在同一行中:

echo -n 'This is stdout (';echo -n stderr >&2 ; echo ')'

--lb 使用 和时,这将被错误地格式化--group

-u但由于进程之间的半行混合,甚至不能保证它会被正确格式化:http://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P

您也可以尝试: --latest-line ,它在屏幕上为每个作业保留一行并在此处打印最新行。

--最新行

答案3

我的解决方案是将输出记录到文件中并使用命令实时观察它的变化tail -f <file>,然后在工作完成时自动删除它们。我还发现--progressflag 很有用。

parallel --progress ./program {} '>' {}.log';' rm {}.log ::: A B C

这里的作业将包括program使用不同的输入运行AB并将C程序的输出发送到相应的日志文件。

相关内容