当天第一次传输 50M 文件时,SFTP put 出现意外 EOF

当天第一次传输 50M 文件时,SFTP put 出现意外 EOF

我正在执行 SFTP 以将文件放入新的外部客户端。我们还有大约 10 个其他客户端,一些是内部的,一些是外部的,我们可以毫无问题地上传到这些客户端。这个新客户端与我们一起使用 VPN(我们的一些其他客户端也与我们一起使用 VPN,但不是全部),在传输 50M 文件中的大约 40M 后,它会以 EOF 响应 put 命令。在过去的 14 次执行中,有 10 次发生了这种情况。

除了两次之外,在当天的第一次运行中,大约凌晨 5:00 时,只收到一次 EOF。几分钟后我再次运行 SFTP 时,put 命令起作用了,并且传输了完整的文件。在当天的剩余时间里,它也起作用了。在同样的执行中,我们还发送了另外两个文件,一个是 1M,另一个是 10M。我的脚本按文件大小对文件列表进行排序,因此较小的文件会先发送。所有这些都在同一个连接中。我的脚本首先连接,发送密码并确保它被接受,然后执行更改目录、ls 命令和 df 命令。然后脚本中有一个循环,用于放置列出的所有文件。前两个文件放置成功,然后放置第三个文件,但因 EOF 而失败。

有两次例外,第二次尝试时我收到了 EOF,第三次尝试时 put 成功。总共,这是最后 14 次运行中的 10 次。其他四次运行在第一次(当天)成功。这项工作大约在同一时间运行(4:50 到 5:15)。我已经与客户进行了交谈,并稍微更改了时间以避开他们运行文件夹监视过程的间隔。但这并没有什么区别。

我们在 AIX 7.1 上运行 OpenSSH_6.0p1,他们在 CentOS Linux 7.4 上运行 OpenSSH_7.4p1。我曾两次尝试使用详细选项 (-vvv) 从命令行运行 SFTP,这两次都成功了。我将更新我的脚本,以某种方式将详细输出捕获到日志文件中。但我在脚本和命令行中都得到了这个 EOF,所以我不认为这是脚本的问题。我可以先发送另外两个文件,所以这不仅仅是发送的第一个文件的问题。这不仅仅是文件大小的问题,即使它只在 50M 文件上失败,因为几分钟后它就可以正常工作了。

所以我的一些问题是 - 什么会导致 put 操作出现 EOF?我找到了 SFTP 的版本 2 RFC 规范,我没有看到关于 put 操作出现 EOF 的提及,只有 get 操作出现 EOF。CentOS Linux 上是否有一些后台进程会导致大文件传输出现 EOF?传输大约需要 25 秒。这可能是网络错误吗?网络质量一开始可能很差,但几分钟后会自动升级到更高质量的水平吗?客户端距离 2700 英里。感谢您的任何评论。Peter

更新:供应商向我发送了他们的 SFTP 调试日志,但我没有发现任何具体原因:

28745   debug1: request 786: write "/upload/X_Y_Z_file_20180315040147.txt" (handle 0) off 25329664 len 32768
28745   debug3: request 786: sent status 0
28745   sent status Success
28745   debug1: request 787: write "/upload/X_Y_Z_file_20180315040147.txt" (handle 0) off 25362432 len 32768
28745   debug3: request 787: sent status 0
28745   sent status Success
28745   debug1: request 788: write "/upload/X_Y_Z_file_20180315040147.txt" (handle 0) off 25395200 len 32768
28745   debug3: request 788: sent status 0
28745   sent status Success
28745   debug1: read eof
28745   forced close "/upload/X_Y_Z_file_20180315040147.txt" bytes read 0 written 25427968
28745   session closed for local user abc from [1.2.3.4]

编辑:我不认为我的脚本是问题所在(但我知道这听起来怎么样),因为无论我使用脚本还是从命令行运行 SFTP 命令,我都会遇到同样的问题。该脚本使用 EXPECT 来捕获 SFTP 响应,该响应是“100%”或 EOF 或 TIMEOUT 之一。一旦我弄清楚如何做到这一点,我就会附上脚本。

编辑:这是 sftp put 的脚本:

#! /usr/bin/ksh
# The following line is seen as a continuecomment by Tcl\
exec $QUOVADX_INSTALL_DIR/integrator/bin/hcitcl "$0" ${1+"$@"}

# subroutines first
proc gts {} { 
   set tt [clock format [clock seconds] -format {%y/%m/%d %H:%M:%S} ]
   set ms [format %03d [expr {[clock clicks -milliseconds] % 1000}]]
   return $tt.$ms 
}

# proc to write to debug file
proc debugw {msg} {
  global debugfile
  set fh [open $debugfile a] ; puts $fh "$msg" ; close $fh
}


set frdir [lindex $argv 0]
set lfile [lindex $argv 1]
set todir [lindex $argv 2]
set site  [lindex $argv 3]

set debug $::env(cldebug)
set parmfile $::env(clparmfile)
set module "sftp_put_list"

# get debugfilename
if {$debug} {set debugfile [exec getdebugfile.tcl] ; global debugfile}
if {$debug} {debugw "[gts] $module start"}
if {$debug} {debugw "[gts] $module processing files in list $lfile"}

# get SFTP connection parms from parmfile using $site as key
set hfile [open $parmfile r] ; set data [read $hfile] ; close $hfile
set lList [split $data "\n"] ; set pos [lsearch -regexp $lList "^key_sftp_$site"]
lassign [split [lindex $lList $pos] " "] key port url pass
if {$debug > 1} {debugw "[gts] $module key: $key port $port url $url pass $pass"}

# url with vertical bar means the first part is the userid and second is the url
set userList [split "$url" "|"] ; lassign $userList user url
if {$debug} {debugw "[gts] $module userList: $userList"}
set ullen [llength $userList]
if {$debug} {debugw "[gts] $module userList length: $ullen"} 


log_user 0

# start/spawn the sftp process - either simple user url or separate user  
if {[llength $userList] eq 1} {
  set hname "$user"
  spawn /usr/bin/sftp -o Port=$port -o ConnectTimeout=60  $user 
  if {$debug} {debugw "[gts] $module connection uses simple user url $user"}
} else {
  set hname "$url"
  spawn /usr/bin/sftp -o Port=$port -o ConnectTimeout=60 -o User=$user $url
  if {$debug} {debugw "[gts] $module connection uses separate user $user and url $url"}
}

set pass_word_prompt "*?assword:*"
set sftp_prompt "*?ftp>"
set timeout 300

set sftp_put_timeout 1000
set prompt_timeout 300

#--------------------------------------------------------------
# first, wait for the password prompt - if a password is required!
#--------------------------------------------------------------
if {$pass ne "#"} {
  expect {
    eof {
       echo "sftp EOF waiting for password prompt|NA"
       if {$debug} {debugw "[gts] $module EOF received waiting for password prompt from $hname - exit 8"}
       exit 8
    }
    "$pass_word_prompt" {
       if {$debug} {debugw "[gts] $module password prompt received and answered"}
    }
    timeout {
       echo "sftp timeout waiting for password prompt|NA"
       if {$debug} {debugw "[gts] $module timeout waiting for password prompt from $hname - exit 8"}
       exit 8
    }
  }

  set timeout $prompt_timeout
  send "$pass\r"
  expect {
    eof {
      echo "sftp EOF waiting for command prompt after entering password|NA"
      if {$debug} {debugw "[gts] $module EOF received waiting for command prompt from $hname after entering password - exit 8"}
      exit 8
    }
    "$sftp_prompt" {
      if {$debug} {debugw "[gts] $module received command prompt; will enter LCD command"}
    }
    timeout {
      echo "sftp timeout after $prompt_timeout seconds waiting for command prompt after entering password|NA"
      if {$debug} {debugw "[gts] $module timeout after $prompt_timeout seconds waiting for command prompt from $hname after entering password - exit 8"}
      exit 8
    }
  }
} else {
  expect {
    eof {
      echo "sftp EOF waiting for command prompt after logging on|NA"
      if {$debug} {debugw "[gts] $module EOF received waiting for command prompt from $hname after logging on - exit 8"}
      exit 8
    }
    "$sftp_prompt" {
      if {$debug} {debugw "[gts] $module received command prompt; will enter LCD command"}
    }
    timeout {
      echo "sftp timeout after $prompt_timeout seconds waiting for command prompt after logging on|NA"
      if {$debug} {debugw "[gts] $module timeout after $prompt_timeout seconds waiting for command prompt from $hname after logging on - exit 8"}
      exit 8
    }
  }

}

#--------------------------------------------------------------
# lcd to local directory from where you will send the file(s)
#--------------------------------------------------------------
set timeout $prompt_timeout
send "lcd $frdir\r"
expect {
   eof {
     echo "sftp EOF waiting for command prompt after entering LCD|NA"
     if {$debug} {debugw "[gts] $module EOF received waiting for command prompt from $hname after entering LCD - exit 8"}
     exit 8
   }
   "$sftp_prompt" {
     if {$debug} {debugw "[gts] $module received command prompt after LCD ; will enter CD command"}
   }
   timeout {
     echo "sftp timeout after $prompt_timeout seconds waiting for command prompt after entering LCD|NA"
     if {$debug} {debugw "[gts] $module timeout after $prompt_timeout seconds waiting for command prompt from $hname after entering LCD - exit 8"}
     exit 8
   }
}

#--------------------------------------------------------------
# cd to remote directory where you will put the file(s)
#--------------------------------------------------------------
set timeout $prompt_timeout
send "cd $todir\r"
expect {
  eof {
    echo "sftp EOF waiting for command prompt after entering CD command|NA"
    if {$debug} {debugw "[gts] $module EOF received waiting for prompt from $hname after entering CD command - exit 8"}
    exit 8
  }
  "No such file or directory" {
    echo "remote directory $todir not found or not accessible|NA"
    if {$debug} {debugw "[gts] $module remote directory $todir not found or not accessible at $hname - exit 8"}
    exit 8
  }
  "$sftp_prompt" {
    if {$debug} {debugw "[gts] $module received command prompt after CD; will execute PWD command"}
  }
  timeout {
    echo "sftp timeout after $prompt_timeout seconds waiting for prompt after entering CD command|NA"
    if {$debug} {debugw "[gts] $module timeout after $prompt_timeou seconds waiting for command prompt from $hname after entering CD command - exit 8"}
    exit 8
  }
}


#--------------------------------------------------------------
# print working directory
#--------------------------------------------------------------
set timeout $prompt_timeout
send "pwd\r"
expect {
  eof {
    echo "sftp EOF waiting for command prompt after entering pwd command|NA"
    if {$debug} {debugw "[gts] $module EOF received waiting for prompt from $hname after entering pwd command - exit 8"}
    exit 8
  }
  "$sftp_prompt" {
    set response $expect_out(0,string)
    set rwd [lindex [split "$response" ":"] 1] 
    set rwd [lindex [split "$rwd" "\r" ] 0]
    if {$debug} {debugw "[gts] $module pwd: $rwd"}
    if {$debug} {debugw "[gts] $module received command prompt after PWD; will execute DIR command"}
  }
  timeout {
    echo "sftp timeout after $prompt_timeout seconds waiting for prompt after entering pwd command|NA"
    if {$debug} {debugw "[gts] $module timeout after $prompt_timeout seconds waiting for command prompt from $hname after entering pwd command - exit 8"}
    exit 8
  }
}

#--------------------------------------------------------------
# directory list
#--------------------------------------------------------------
set timeout $prompt_timeout
send "ls -l\r"
expect {
  eof {
    echo "sftp EOF waiting for command prompt after entering dir command|NA"
    if {$debug} {debugw "[gts] $module EOF received waiting for prompt from $hname after entering dir command - exit 8"}
    exit 8
  }
  "$sftp_prompt" {
    set response $expect_out(0,string)
    set dir [lrange [split "$response" "\r"] 1 end-1] 
    set dir [string map {\{ "" \} ""} "$dir"]
    set dir [string trim "$dir"]
    if {$debug} {debugw "[gts] $module dir: \r$dir"}
    if {$debug} {debugw "[gts] $module received command prompt after DIR; will execute PUT command"}
  }
  timeout {
    echo "sftp timeout after $prompt_timeout seconds waiting for prompt after entering dir command|NA"
    if {$debug} {debugw "[gts] $module timeout after $prompt_timeout seconds waiting for command prompt from $hname after entering dir command - exit 8"}
    exit 8
  }
}

#------------------------------------------------
# loop through list of files - put each file to remote sftp server
#------------------------------------------------
set hfile [open $lfile r] ; set data [read $hfile] ; close $hfile
set flist [string map {"\n" " "} "$data" ] ; set flist [string trim "$flist"] 
set flist [split $flist " "] ; set nfiles [llength $flist]
if {$debug} {debugw "[gts] $module $nfiles file names found in list file $lfile"}

set ct 0
while {$ct < $nfiles} {
  set nextfile [lindex $flist $ct]
  if {$debug} {debugw "[gts] $module starting SFTP Put for filename $nextfile"}
  set dct [expr $ct + 1]

  set timeout $sftp_put_timeout
  send "put $nextfile\r"
  expect {
    eof {
      set response $expect_out(0,string)
      echo "sftp EOF waiting for command prompt after entering put command|$nextfile"
      if {$debug} {debugw "[gts] $module put file $dct - $nextfile failed"}
      if {$debug} {debugw "[gts] $module EOF response from Put:\n$response"}
      if {$debug} {debugw "[gts] $module EOF received waiting for prompt from $hname after entering put command - exit 8"}
      exit 8
    }
    "100%*$sftp_prompt" {
      if {$debug} {debugw "[gts] $module received PUT 100% and command prompt after PUT"}
      if {$debug} {debugw "[gts] $module put file $dct - $nextfile succeeded"}
    }
    timeout {
      echo "sftp timeout after $sftp_put_timeout seconds waiting for prompt after entering put command|$nextfile"
      if {$debug} {debugw "[gts] $module timeout after $sftp_put_timeout seconds waiting for command prompt from $hname after entering put command - exit 8"}
      exit 8
    }
  }

  incr ct
}

#--------------------------------------------------------------
# post transfer directory list
#--------------------------------------------------------------
set timeout $prompt_timeout
send "ls -l\r"
expect {
  eof {
    echo "sftp EOF waiting for command prompt after entering dir command|NA"
    if {$debug} {debugw "[gts] $module EOF received waiting for prompt from $hname after entering dir command - exit 8"}
    exit 8
  }
  "$sftp_prompt" {
    set response $expect_out(0,string)
    set dir [lrange [split "$response" "\r"] 1 end-1] 
    set dir [string map {\{ "" \} ""} "$dir"]
    set dir [string trim "$dir"]
    if {$debug} {debugw "[gts] $module dir: \r$dir"}
    if {$debug} {debugw "[gts] $module received command prompt after dir; will execute QUIT command"}
  }
  timeout {
    echo "sftp timeout after $prompt_timeout seconds waiting for prompt after entering dir command|NA"
    if {$debug} {debugw "[gts] $module timeout after $prompt_timeout seconds waiting for command prompt from $hname after entering dir command - exit 8"}
    exit 8
  }
}

#---------------------------------------------------------
# sftp checks for spawn EOF and will terminate prematurely
# unless you add extra expect command
#---------------------------------------------------------
set timeout $prompt_timeout
send "quit\r"
expect {
  eof {
    if {$debug} {debugw "[gts] $module EOF expected and received after QUIT command|NA"}
    exit 0
  }
  "$sftp_prompt" {
    echo "unexpected sftp prompt after entering QUIT command - should be EOF|NA"
    if {$debug} {debugw "[gts] $module unexpected sftp prompt after entering QUIT command - should be EOF - exit 2"}
    exit 2
  }
  timeout {
    echo "sftp timeout after $prompt_timeout seconds waiting for EOF after QUIT command|NA"
    if {$debug} {debugw "[gts] $module sftp timeout after $prompt_timeout seconds waiting for EOF after QUIT command - exit 8"}
    exit 8
  }
}

exit 0

相关内容