☰

如何在unix中对发出post请求的函数进行多个并行调用？

tag-icon • tag-icon bash shell-script shell curl parallelism
如何在unix中对发出post请求的函数进行多个并行调用？

我有一个函数 A，它接受一个参数 fileName，然后使用该文件对就业服务器进行curl post 调用。该函数伪代码将如下所示：

function A(filename)
{
// read file from local 
// send it to deployment server and via curl/any rest client
// read response code
// store data of filename and response code 
}

for all files f in folder :
 A(f)

// See all response code and print data, which all files passed and which all failed

现在，如果我想并行运行，我将使用：

A(f) & inside for loop

我担心突出显示的部分，如何收集文件名、响应代码的详细信息，因为 unix 中没有容易获得的哈希图/字典类型的支持。另外，如果我打印 A 中的数据，那么如果我保留两行，一行打印文件名，另一行打印响应代码，那么由于并行性质，两行可能不是连续的。虽然我可以在一行中打印两者，但我想捕获结果以便稍后进行一些处理。有没有一种简单的方法？

答案1

我会用parset：

a() {
  filename="$1"
  echo "***  Here    is  ***"
  sleep 0.$RANDOM
  echo "the response code to"
  sleep 0.$RANDOM
  echo "$filename"
}
export -f a

# Put the results into a bash array. The order will be kept.
# Here we use "File  a" and "File  b" as input. Change those to your needs
parset res a ::: "File  a" "File  b"
echo "${res[0]}"
echo "${res[1]}"

或者：

# If you want the output into an assoc array, convert the results:
# First build an array of the input
input=("File  a" "File  b")
# Then run the jobs in parallel
parset res a ::: "${input[@]}"

# Finally zip the two arrays to a single associative array
declare -A myassoc
for ((i=0; $i<${#input[@]}; i++)); do
  myassoc[${input[i]}]=${res[i]}
done
echo "${myassoc["File  a"]}"

parset是 GNU Parallel 的一部分。

要激活parset您需要将其作为功能激活。该函数被定义为 的一部分env_parallel。

执行以下操作并重新启动 shell。

bash:  Put this in $HOME/.bashrc:  . `which env_parallel.bash`
       E.g. by doing:  echo '. `which env_parallel.bash`' >> $HOME/.bashrc
       Supports: aliases, functions, variables, arrays

zsh:   Put this in $HOME/.zshrc:  . `which env_parallel.zsh`
       E.g. by doing:  echo '. `which env_parallel.zsh`' >> $HOME/.zshenv
       Supports: functions, variables, arrays

fish:  Unsupported

ksh:   Put this in $HOME/.kshrc:  source `which env_parallel.ksh`
       E.g. by doing:  echo 'source `which env_parallel.ksh`' >> $HOME/.kshrc
       Supports: aliases, functions, variables, arrays

mksh:  Put this in $HOME/.mkshrc:  source `which env_parallel.mksh`
       E.g. by doing:  echo 'source `which env_parallel.mksh`' >> $HOME/.mkshrc
       Supports: aliases, functions, variables, arrays

pdksh: Put this in $HOME/.profile:  source `which env_parallel.pdksh`
       E.g. by doing:  echo '. `which env_parallel.pdksh`' >> $HOME/.profile
       Supports: aliases, functions, variables, arrays

ash:   Put this in $HOME/.profile:  . `which env_parallel.ash`
       E.g. by doing:  echo '. `which env_parallel.ash`' >> $HOME/.profile
       Supports: aliases, variables

dash:  Put this in $HOME/.profile:  . `which env_parallel.dash`
       E.g. by doing:  echo '. `which env_parallel.dash`' >> $HOME/.profile
       Supports: aliases, variables

csh:   Unsupported

tcsh:  Unsupported

To install in all shells run:

  parset --install

相关内容