![如何发送新命令运行到已经运行的 nohup 进程或在 nohup 中一起/同时运行两个命令?](https://linux22.com/image/213332/%E5%A6%82%E4%BD%95%E5%8F%91%E9%80%81%E6%96%B0%E5%91%BD%E4%BB%A4%E8%BF%90%E8%A1%8C%E5%88%B0%E5%B7%B2%E7%BB%8F%E8%BF%90%E8%A1%8C%E7%9A%84%20nohup%20%E8%BF%9B%E7%A8%8B%E6%88%96%E5%9C%A8%20nohup%20%E4%B8%AD%E4%B8%80%E8%B5%B7%2F%E5%90%8C%E6%97%B6%E8%BF%90%E8%A1%8C%E4%B8%A4%E4%B8%AA%E5%91%BD%E4%BB%A4%EF%BC%9F.png)
我想运行 nohup 作业,并对已经运行的进程运行特定的新命令(例如 kerberos 身份验证)。事实上,理想的解决方案是首先运行 reauth 命令,然后运行真正的作业都在同一个 nohup 进程 id 下。这样 nohup 进程就不会丢失 kerberos 票证。所以我想与另一个 python 脚本同步运行 reauth,这样就不会丢失他们的 kerberos 票证——没有屏幕或 tmux或任何需要我与脚本交互的事情。我基本上想实现一个运行 python 作业(或任何作业)的守护进程,在运行完成之前永远不会丢失它的票证。如何做到这一点?
这是我目前的尝试,但我不太确定它是否有效
# - set up this main sh script
source ~/.bashrc
source ~/.bash_profile
source ~/.bashrc.user
echo HOME = $HOME
source cuda11.1
conda init bash
conda activate metalearning_gpu
# - get a job id for this tmux session
export SLURM_JOBID=$(python -c "import random;print(random.randint(0, 1_000_000))")
echo SLURM_JOBID = $SLURM_JOBID
export OUT_FILE=$PWD/main.sh.o$SLURM_JOBID
export ERR_FILE=$PWD/main.sh.e$SLURM_JOBID
export WANDB_DIR=$HOME/wandb_dir
export CUDA_VISIBLE_DEVICES=4
echo CUDA_VISIBLE_DEVICES = $CUDA_VISIBLE_DEVICES
python -c "import torch; print(torch.cuda.get_device_name(0));"
# - CAREFUL, if a job is already running it could do damage to it, rm reauth process, qian doesn't do it so skip it
# top -u brando9
# pkill -9 reauth -u brando9
# - expt python script then inside that python pid attach a reauth process
# should I run rauth within python with subprocess or package both the nohup command and the rauth together in badsh somehow
#python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE &
#nohup (echo $SU_PASSWORD | /afs/cs/software/bin/reauth; python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE) &
nohup (echo $SU_PASSWORD | /afs/cs/software/bin/reauth; python -u ~/main.py > $OUT_FILE 2> $ERR_FILE) &
# other option is to run `echo $SU_PASSWORD | /afs/cs/software/bin/reauth` inside of python, right?
export JOB_PID=$!
echo JOB_PID = $JOB_PID
# - Done
echo "Done with the dispatching (daemon) sh script
可能是因为它运行的时间不够长?没有把握。
我认为另一种选择是在 python 脚本本身内部运行 reauth。没有测试过但可能有用吗?主要技巧是不要像我的 nohup 进程那样由于丢失 kerberos 票证而死掉。
reauth的内容是:
(metalearning_gpu)~ $ cat /afs/cs/software/bin/reauth
#!/usr/bin/perl
# $Id: reauth 2737 2011-06-20 18:14:05Z miles $
#
# Original version (C) Martin Schulz, 2'2002
# University Karlsruhe
#
# Modifications by Miles Davis <[email protected]>
# Super minimal -- call programs rather than functions to reduce dependence
# on extra perl modules.
#
# Heimdal patches thanks to Georgios Asimenos <[email protected]>
#
# General:
##########
# This little script aims at maintaining a valid AFS token from a
# users password for long running jobs.
# As everybody knows (or should know) Kerberos tickets and AFS tokens
# only have a limited lifetime. This is so by design and is usually
# reasonable. After 12 hours, it is no more obvious that it is really
# that user sitting in front of the computer that once typed the
# correct password in. Furthermore the damage caused by compromized
# AFS tokens is limited to the lifetime of that ticket.
# However, there are situations when users want to use long running
# jobs that will write to AFS filespace for several days. Renewable
# tickets are not so much of help here, since they can only be renewed
# if ....
# Therefore the secret has somehow deposited on the local computer
# that will run the long time job. This can be eiter done by storing a
# keytab on the local disk, maybe with a cron(*) principal with
# reduces priviledges. The approach taken here is to work with the
# original password and keep it in RAM only.
# When starting this program, the user is asked for his principal and
# the corresponding password. Then the TGT and AFS token is obtained
# and displayed, afterwards, a background process is forked and the
# main process will return to the system prompt. The workload program
# can now be started.
# The background process will periodically attempt to obtain krb
# tickets and AFS tokens. If this fails for some reason (Kerberos
# server not available or anything, the program aborts.
# aklog does not create a new pag if not told so. If you want your
# background process have a separate pag, create it beforehand.
# The reauth.pl program will work until eternity if is not stopped
# somehow. The canonical way is kill it by "kill $pid", where $pid is
# the process id printed before the return of the initial call to
# reauth.pl or found in the output of "ps".
# (*) Cron jobs are another issue. Our institute introduced
# user.cron-style principals to enable cron to obtain a token and then
# work on restricted parts of the users home directories.
# Security issues:
##################
# reauth.pl will run forever if you do not stop it, so don't forget that!
# The password is kept in RAM (of the child process). AFAIK, this can
# only be recovered by local root (who you need to trust anyway). It
# will not survive a reboot of the local machine.
# The password is not kept on any disk. Therefore any bootfloppy
# (reboot to single user mode..) or screwdriver (take disk away..)
# attacks are not promising.
# Be aware that your NSA-, FBI-, MI5-, KGB-, ElQaida-, or (*insert
# your favorite opponent or competitor here*)-sponsored cleaning
# personnel or coworkers might have even more elaborate means... :-)
# BUGS:
#######
# Only mildly tested only on Linux and Solaris.
# Uses kinit, aklog, klist and tokens programs for a KerberosV/ Ken
# Hornstein's migration kit centered AFS setup. Please adjust to your
# config.
###########################################################################
# Configs:
# kinit program, add path if necessary
if ( -e "/usr/kerberos/bin/kinit" ) {
$kinit="/usr/kerberos/bin/kinit";
} elsif ( -e "/usr/lib/heimdal/bin/kinit" ) {
$kinit = "/usr/lib/heimdal/bin/kinit";
} elsif ( -e "/usr/bin/kinit" ) {
$kinit="/usr/bin/kinit";
} else {
die("Couln't find kinit.\n");
}
# aklog program, add path if necessary
if ( -e "/usr/bin/aklog" ) {
$aklog="/usr/bin/aklog";
} elsif ( -e "/usr/lib/heimdal/bin/afslog" ) {
# or, afslog, for heimdal weirdos
$aklog="/usr/lib/heimdal/bin/afslog";
} else {
die("Couln't find aklog or afslog.\n");
}
# klist program, add path if necessary
$klist="/usr/kerberos/bin/klist";
# tokens program, add path if necessary
$tokens="/usr/bin/tokens";
#################################################################
# Program:
use Getopt::Long;
use POSIX qw(setuid);
use POSIX qw(setgid);
use POSIX qw(setsid);
# Defaults for command line options.
my $keytab = '';
my $command = '';
my $username = '';
my $debug = 0;
my $verbose = 0;
my $interval=15000; # time interval in seconds: 4+ hours:
my %opts = (
# Keytab
'k=s' => sub {
$keytab = @_[1];
$kinit_opts .= "-k -t $keytab ";
},
# Run command
'c=s' => sub {
$command = @_[1];
},
# Run command as user
'u=s' => sub {
$username = @_[1];
},
# Time interval to sleep
'i=i' => sub {
$interval = @_[1];
},
# Debug
'd' => sub {
$debug++;
},
# Be versbose
'v' => sub {
$verbose++;
},
);
GetOptions(%opts) or die "Usage: reauth [ -k=keytab ] [ -u user ] [ -i <sleep_interval ] [ -v ] [ -c <command> ]\n";
if(@ARGV) {
$princ = $ARGV[0];
debug_print(2, "Principal name provided by argument = $princ");
} else {
# Assume we want the login name as the principal name
$princ = getpwuid($<);
debug_print(2, "Principal name provided by argument = $princ");
}
if ($keytab) {
# Don't ask for password, a keytab was provided.
debug_print(1, "Keytab provided = $keytab");
} else {
# read password, but turn off echo before:
print "Password for $princ: ";
system "stty -echo";
$passwd = <STDIN>;
system "stty echo";
printf "\n";
chomp $passwd;
# Actually get the tickets/tokens
if(obtain_tokens()!=0) {
die "Can't obtain kerberos tickets\n";
}
if ($verbose) {
show_tokens();
}
}
# fork to go into background:
# a) the parent will exit
# b) the child will work on
$pid = fork();
if ($pid) {
# I am the parent.
printf "Background process pid is: $pid\n";
if ($command) {
debug_print(1,"Waiting for child to die.");
wait;
debug_print(1,"Child is dead.");
}
exit 0;
} else {
# I am the child.
debug_print(2,"I am process $$");
print "Can't set session id\n" unless setsid();
debug_print(2,"KRB5CCNAME: " . $ENV{KRB5CCNAME});
#if ($ENV{KRB5CCNAME}) {
#$ENV{KRB5CCNAME} = $ENV{KRB5CCNAME} . "_reauth_$$";
#} else {
#$ENV{KRB5CCNAME} = "/tmp/krb5cc_reauth_$$";
#}
#debug_print(2,"Creating " . $ENV{KRB5CCNAME});
#system "touch $ENV{KRB5CCNAME}";
if ($username) {
debug_print(1, "Looking up UID for $username");
($name,$passwd,$UID,$GID, @junk) = getpwnam($username);
debug_print(1, "Changing to UID $UID, GID $GID");
print "Can't set group id\n" unless setgid($GID);
print "Can't set user id\n" unless setuid($UID);
if ($ENV{KRB5CCNAME}) {
$ENV{KRB5CCNAME} = $ENV{KRB5CCNAME} . "_reauth_$$";
} else {
$ENV{KRB5CCNAME} = "/tmp/krb5cc_reauth_$$";
}
}
debug_print(2, "Running as uid " . $<);
# Actually get the tickets/tokens
if(obtain_tokens()!=0) {
die "Can't obtain kerberos tickets\n";
}
if ($verbose) {
show_tokens();
}
# If I was told to run a command, do it.
if ($command) {
debug_print(1,"About to exec $command");
exec($command) or die "Can't execute '$command'.\n";
exit
}
debug_print(2,"Going into auth loop (interval is $interval).");
#close(STDOUT);
#close(STDERR);
# Otherwise, work until killed:
while (1) {
debug_print(2,"Waking up to obtain new tokens.");
obtain_tokens();
if ($verbose) {
show_tokens();
}
sleep $interval;
};
}
#################################################################
sub obtain_tokens() {
# ignore sigpipes' (according to perlopentut)
$SIG{PIPE} = 'IGNORE';
#debug_print(1,"Running: | $kinit -f $kinit_opts -p $princ 1>/dev/null 2>&1");
# run kinit
open(KINIT, "| $kinit -f $kinit_opts -p $princ 1>/dev/null 2>&1");
# pass password to stdin, password does not show up on command line
if (! $keytab) {
print(KINIT "${passwd}\n");
}
# close pipe and get status
close(KINIT); $status=$?;
debug_print(1,"kinit exited with status $status\n");
# act on status..
if($status == 256) {
if ($verbose) {
print "WARNING: kinit is not able to obtain Kerberos ticket ($status).\n";
print " Possible DNS or network problem. Continuing anyway...\n";
}
return 1;
} elsif($status!=0) {
print "kinit is not able to obtain Kerberos ticket: $status\n";
return 2;
};
debug_print(1,"Running $aklog...\n");
$status = system "$aklog >/dev/null" ;
debug_print(1,"aklog exited with status $status\n");
if($status!=0) {
print "aklog is not able to obtain AFS token: $status\n";
return 3;
};
return 0;
};
##################################################################
sub show_tokens() {
system $klist ;
system $tokens ;
};
##################################################################
sub debug_print($$) {
my $level = shift;
my $message = shift;
if ($debug >= $level) {
print "DEBUG$debug: $message\n";
}
}
##################################################################
python 重新验证尝试:
def run_bash_command(cmd: str) -> Any:
import subprocess
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
if error:
raise Exception(error)
else:
return output
def stanford_reauth():
# def stanford_rauth(password: Optional[str] = None):
# password: str = os.environ['SU_PASSWORD'] if password is None else None
# assert password is not None, f'Err: {password=}'
reauth_cmd: str = f'echo $SU_PASSWORD | /afs/cs/software/bin/reauth'
out = run_bash_command(reauth_cmd)
print('Output of reauth (/afs/cs/software/bin/reauth with password)')
print(f'{out=}')
我目前正在运行两者,看看它们是否通过/没有被杀死。
新错误:
我正在尝试我的尝试:
nohup sh -c "echo $SU_PASSWORD | /afs/cs/software/bin/reauth; python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_diversity_with_task2vec.py --manual_loads_name diversity_ala_task2vec_hdb1_mio > $OUT_FILE 2> $ERR_FILE" > $PWD/main.sh.nohup.out$SLURM_JOBID &
但我不确定它是否还有效。我收到以下错误:
Password for brando9: stty: 'standard input': Inappropriate ioctl for device
stty: 'standard input': Inappropriate ioctl for device
Can't obtain kerberos tickets
我想我明白这个问题了。新命令希望终端发送密码,因此它不会让 echo 发送密码。我试过了:
- 期望(但没有 sudo 或安装它的方法)
- 这里的大部分东西https://serverfault.com/questions/241588/how-to-automate-ssh-login-with-password但不能做 sudo 来最正确地尝试。
我确实有 ssh 和需要发送密码。有没有办法用 发送密码ssh
?
有关的:
- 红迪十字:https://www.reddit.com/r/unix/comments/yvn5t3/how_does_one_send_new_commands_to_run_to_an/
- 法定交叉:https://www.quora.com/unanswered/How-does-one-send-new-commands-to-run-to-an-already-running-nohup-process-or-run-two-commands-together-并发 in nohup
- linux 红迪网:https://www.reddit.com/r/linuxquestions/comments/yvo8ti/how_does_one_send_new_commands_to_run_to_an/
- 相关但与 tmux (或我认为屏幕没问题):如何使用 tmux 在后台运行作业而不与 tmux 交互(例如分派 nohup 作业)?
- 有关的:如何等待进程结束然后执行一系列操作(例如终止 tmux 会话)而不阻塞我的主终端?
- 有关的,在 Linux 中如何使用需要密码的命令进行身份验证?
- 有关的:当您不是 tty/stty(终端)并且没有 Expect、sshpass 时,如何向命令发送密码?
答案1
我不知道为什么,但这似乎停止工作这里询问的替代方法:当您不是 tty/stty(终端)并且没有 Expect、sshpass 时,如何向命令发送密码?
看来 3 次尝试都有效:
- 在引用的 sh 脚本中运行 reauth
- 在 python 中运行 reauth
- 执行上述两项操作
我用来运行作业的示例脚本:
# https://unix.stackexchange.com/questions/724902/how-does-one-send-new-commands-to-run-to-an-already-running-nohup-process-e-g-r
# sh ~/diversity-for-predictive-success-of-meta-learning/main_nohup_snap.sh
# - set up this main sh script
export RUN_PWD=$(pwd)
source ~/.bashrc
source ~/.bash_profile
source ~/.bashrc.user
echo HOME = $HOME
# since snap .bash.user cd's me into HOME at dfs
cd $RUN_PWD
echo RUN_PWD = $RUN_PWD
realpath .
source cuda11.1
conda init bash
conda activate metalearning_gpu
# - get a job id for this tmux session
export SLURM_JOBID=$(python -c "import random;print(random.randint(0, 1_000_000))")
echo SLURM_JOBID = $SLURM_JOBID
export OUT_FILE=$PWD/main.sh.o$SLURM_JOBID
export ERR_FILE=$PWD/main.sh.e$SLURM_JOBID
export WANDB_DIR=$HOME/wandb_dir
echo $OUT_FILE
echo $ERR_FILE
export CUDA_VISIBLE_DEVICES=5
echo CUDA_VISIBLE_DEVICES = $CUDA_VISIBLE_DEVICES
python -c "import torch; print(torch.cuda.get_device_name(0));"
# - CAREFUL, if a job is already running it could do damage to it, rm reauth process, qian doesn't do it so skip it
# top -u brando9
# pkill -9 reauth -u brando9
# - expt python script then inside that python pid attach a reauth process
# should I run rauth within python with subprocess or package both the nohup command and the rauth together in badsh somehow
#python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE &
nohup sh -c 'echo $SU_PASSWORD | /afs/cs/software/bin/reauth; python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE' > $PWD/nohup.out$SLURM_JOBID &
#nohup python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE &
# other option is to run `echo $SU_PASSWORD | /afs/cs/software/bin/reauth` inside of python, right?
export JOB_PID=$!
echo JOB_PID = $JOB_PID
echo SLURM_JOBID = $SLURM_JOBID
# - Done
echo "Done with the dispatching (daemon) sh script"
主要命令:
nohup sh -c 'echo $SU_PASSWORD | /afs/cs/software/bin/reauth; python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE' > $PWD/nohup.out$SLURM_JOBID &
我想我会坚持使用选项 3,所以运行两者,因为如果我运行分布式作业,我不希望这些进程在未经我许可的情况下被随机终止。为此,Python 脚本:
def run_bash_command(cmd: str) -> Any:
import subprocess
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
if error:
raise Exception(error)
else:
return output
def stanford_reauth():
""""
re-authenticates the python process in the kerberos system so that the
python process is not killed randomly.
ref: https://unix.stackexchange.com/questions/724902/how-does-one-send-new-commands-to-run-to-an-already-running-nohup-process-or-run
"""
reauth_cmd: str = f'echo $SU_PASSWORD | /afs/cs/software/bin/reauth'
out = run_bash_command(reauth_cmd)
print('Output of reauth (/afs/cs/software/bin/reauth with password): ')
print(f'{out=}')
不要忘记经常检查 reauth 杀死 reauth 作业:
# - CAREFUL, if a job is already running it could do damage to it, rm reauth process
# pkill -9 reauth -u brando9