我有一个文件存储一些命令,如下所示:
sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar
仅当附加了文件名(例如 )时,上述命令才会运行file.py
。所以,如果我要在命令行中运行它,我会输入
sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar file.py
但是,我想使用此命令运行一些文件,并且我想在每次运行它时指定文件名。我尝试将此命令保存在名为 的文件中command
,并且执行了
cat command echo file.py | bash
但它似乎不起作用。我应该如何继续?
答案1
在您的脚本中,只需将其设置为一个变量并进行快速健全性检查即可:
pyfile="file.py"
sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar "${pyfile?python script not specified}"
如果变量未定义或为空,该构造${var?message}
将抛出错误并显示。message
var
您也可以使用 提供默认值${var-defaultvalue}
。
您还可以将其设置为更简单的调用函数:
runjob() {
sudo PYSPARK_DRIVER_PYTHON=/bin/python2.7 PYSPARK_PYTHON=/bin/python2.7 SPARK_CONF_DIR=/configuration/spark2 /spark2.1/bin/spark-submit --driver-memory 2g --executor-memory 4g --num-executors 100 --jars /lib/json-serde-1.3.7-jar-with-dependencies.jar "${1?python script not specified}"
}
runjob "/path/to/file.py"
runjob "/path/to/some/other/file.py"