☰

2. 我为什么会得到这个信息？

tag-icon • tag-icon ubuntu openvpn
2. 我为什么会得到这个信息？
  • Ubuntu 14.04
  • OpenVPN 2.3.2
  • 初始化守护进程：upstart

服务器配置文件：

topology net30
mode server
tls-server
push "route-gateway 192.168.0.1"
ifconfig 192.168.0.1 192.168.0.2
ifconfig-pool 192.168.0.5 192.168.0.9

port 1196
proto udp
dev tun
keepalive 10 120
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status /var/log/openvpn/devops.log 1
syslog vpn-devops
verb 15
writepid /var/run/openvpn/devops.pid
ca /etc/openvpn/devops/ca.crt
cert /etc/openvpn/devops/server.crt
key /etc/openvpn/devops/server.key
dh /etc/openvpn/dh.pem
ifconfig-pool-persist /var/lib/openvpn/devops.ipp
client-config-dir /etc/openvpn/devops/ccd
crl-verify /etc/openvpn/devops/crl.pem
client-to-client

问题：每当重新启动服务时，我都会在日志文件中看到以下内容：

May 17 14:32:08 t-xhyve vpn-devops[21013]: TCP/UDP: Closing socket
May 17 14:32:08 t-xhyve vpn-devops[21013]: Closing TUN/TAP interface
May 17 14:32:08 t-xhyve vpn-devops[21013]: /sbin/ip addr del dev tun1 local 192.168.0.1 peer 192.168.0.2
May 17 14:32:08 t-xhyve vpn-devops[21013]: Linux ip addr del failed: external program exited with error status: 2

原因是 OpenVPN 以nobody无权删除隧道接口的用户身份运行。因此，我想使用根深蒂固插件以root权限调用down脚本。

plugin /usr/lib/openvpn/openvpn-plugin-down-root.so /usr/share/openvpn/down.sh

/usr/share/openvpn/down.sh

#!/bin/sh

/sbin/ip addr del dev $1 local $4 peer $5

然后尝试重新启动但我收到此信息：

May 17 14:49:47 t-xhyve vpn-devops[23001]: /sbin/ip addr del dev tun1 local 192.168.0.1 peer 192.168.0.2
May 17 14:49:47 t-xhyve vpn-devops[23228]: PKCS#11: __pkcs11h_forkFixup entry pid=23228, activate_slotevent=1
May 17 14:49:47 t-xhyve vpn-devops[23228]: PKCS#11: __pkcs11h_forkFixup return
May 17 14:49:47 t-xhyve vpn-devops[23001]: Linux ip addr del failed: external program exited with error status: 2
May 17 14:49:47 t-xhyve vpn-devops[23001]: PLUGIN_CALL: PRE type=PLUGIN_DOWN
May 17 14:49:47 t-xhyve vpn-devops[23001]: ARGV[0] = '/usr/lib/openvpn/openvpn-plugin-down-root.so'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ARGV[1] = 'tun1'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ARGV[2] = '1500'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ARGV[3] = '1542'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ARGV[4] = '192.168.0.1'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ARGV[5] = '192.168.0.2'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ARGV[6] = 'init'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[0] = 'dev=tun1'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[1] = 'link_mtu=1542'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[2] = 'tun_mtu=1500'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[3] = 'script_context=init'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[4] = 'signal=sigterm'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[5] = 'redirect_gateway=0'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[6] = 'dev_type=tun'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[7] = 'ifconfig_remote=192.168.0.2'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[8] = 'ifconfig_local=192.168.0.1'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[9] = 'config=/etc/openvpn/devops/config'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[10] = 'verb=15'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[11] = 'daemon=0'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[12] = 'daemon_log_redirect=0'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[13] = 'daemon_start_time=1463471367'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[14] = 'daemon_pid=23001'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[15] = 'proto_1=udp'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[16] = 'local_port_1=1196'
May 17 14:49:47 t-xhyve vpn-devops[23001]: ENVP[17] = 'remote_port_1=1196'
May 17 14:49:47 t-xhyve vpn-devops[23001]: PLUGIN_CALL: POST /usr/lib/openvpn/openvpn-plugin-down-root.so/PLUGIN_DOWN status=1
May 17 14:49:47 t-xhyve vpn-devops[23001]: PLUGIN_CALL: plugin function PLUGIN_DOWN failed with status 1: /usr/lib/openvpn/openvpn-plugin-down-root.so
May 17 14:49:47 t-xhyve vpn-devops[23001]: ERROR: up/down plugin call failed
May 17 14:49:47 t-xhyve vpn-devops[23001]: Exiting due to fatal error
May 17 14:49:50 t-xhyve openvpn-devops-upstart: RTNETLINK answers: Operation not permitted
May 17 14:49:50 t-xhyve openvpn-devops-upstart: DOWN-ROOT: Error sending script execution signal to background process

我有一些疑问：

  1. 为什么 OpenVPN 仍这样称呼？

    /sbin/ip addr del dev tun1 local 192.168.0.1 peer 192.168.0.2

  2. 我为什么会得到这个？

    May 17 14:49:47 t-xhyve vpn-devops[23001]: ERROR: up/down plugin call failed
May 17 14:49:47 t-xhyve vpn-devops[23001]: Exiting due to fatal error
May 17 14:49:50 t-xhyve openvpn-devops-upstart: RTNETLINK answers: Operation not permitted
May 17 14:49:50 t-xhyve openvpn-devops-upstart: DOWN-ROOT: Error sending script execution signal to background process

如何解决？

是否有任何适当的应用程序保护配置文件？

不。

尝试修改 shell 脚本来执行 cat /proc/self/status >/tmp/temp.txt 并在此处打印出结果。

/usr/share/openvpn/down.sh

#!/bin/sh

cat /proc/self/status >/tmp/temp.txt
#/sbin/ip addr del dev $1 local $4 peer $5

重新启动时：

May 18 16:51:00 t-xhyve vpn-devops[17930]: TCP/UDP: Closing socket
May 18 16:51:00 t-xhyve vpn-devops[17930]: Closing TUN/TAP interface
May 18 16:51:00 t-xhyve vpn-devops[17930]: /sbin/ip addr del dev tun1 local 192.168.0.1 peer 192.168.0.2
May 18 16:51:00 t-xhyve vpn-devops[18520]: PKCS#11: __pkcs11h_forkFixup entry pid=18520, activate_slotevent=1
May 18 16:51:00 t-xhyve vpn-devops[18520]: PKCS#11: __pkcs11h_forkFixup return
May 18 16:51:00 t-xhyve vpn-devops[17930]: Linux ip addr del failed: external program exited with error status: 2
May 18 16:51:00 t-xhyve vpn-devops[17930]: PLUGIN_CALL: PRE type=PLUGIN_DOWN
May 18 16:51:00 t-xhyve vpn-devops[17930]: ARGV[0] = '/usr/lib/openvpn/openvpn-plugin-down-root.so'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ARGV[1] = 'tun1'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ARGV[2] = '1500'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ARGV[3] = '1542'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ARGV[4] = '192.168.0.1'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ARGV[5] = '192.168.0.2'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ARGV[6] = 'init'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[0] = 'dev=tun1'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[1] = 'link_mtu=1542'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[2] = 'tun_mtu=1500'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[3] = 'script_context=init'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[4] = 'signal=sigterm'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[5] = 'redirect_gateway=0'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[6] = 'dev_type=tun'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[7] = 'ifconfig_remote=192.168.0.2'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[8] = 'ifconfig_local=192.168.0.1'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[9] = 'config=/etc/openvpn/devops/config'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[10] = 'verb=15'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[11] = 'daemon=0'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[12] = 'daemon_log_redirect=0'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[13] = 'daemon_start_time=1463564660'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[14] = 'daemon_pid=17930'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[15] = 'proto_1=udp'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[16] = 'local_port_1=1196'
May 18 16:51:00 t-xhyve vpn-devops[17930]: ENVP[17] = 'remote_port_1=1196'
May 18 16:51:00 t-xhyve vpn-devops[17930]: PLUGIN_CALL: POST /usr/lib/openvpn/openvpn-plugin-down-root.so/PLUGIN_DOWN status=1
May 18 16:51:00 t-xhyve vpn-devops[17930]: PLUGIN_CALL: plugin function PLUGIN_DOWN failed with status 1: /usr/lib/openvpn/openvpn-plugin-down-root.so
May 18 16:51:00 t-xhyve vpn-devops[17930]: ERROR: up/down plugin call failed
May 18 16:51:00 t-xhyve vpn-devops[17930]: Exiting due to fatal error
May 18 16:51:00 t-xhyve [2016-05-18 16:51:00,738][WARN ]May 18 16:51:01 t-xhyve openvpn-devops-upstart: RTNETLINK answers: Operation not permitted
May 18 16:51:01 t-xhyve openvpn-devops-upstart: DOWN-ROOT: Error sending script execution signal to background process

该过程重新启动，但日志文件 ( /tmp/temp.txt) 未创建。

当然，手动运行该脚本可以正常工作：

bash -x /usr/share/openvpn/down.sh
 + cat /proc/self/status
 + 

cat /tmp/temp.txt
Name:   cat
State:  R (running)
Tgid:   19246
Ngid:   0
Pid:    19246
PPid:   19245
...

指定完整路径没有帮助：

May 19 03:52:11 t-xhyve openvpn-devops-upstart: RTNETLINK answers: Operation not permitted
May 19 03:52:11 t-xhyve openvpn-devops-upstart: DOWN-ROOT: Error sending script execution signal to background process
May 19 03:52:11 t-xhyve openvpn-devops-upstart: DOWN-ROOT: BACKGROUND: INIT command='/bin/cat /proc/self/status > /tmp/temp.txt'

您如何启动/停止服务？

@rda：这是 upstart 文件：

start on (net-device-up
          and local-filesystems
          and runlevel [2345]
          and started rsyslog)
stop on runlevel [!2345]

respawn
respawn limit 6 60

pre-start script
    if [ ! -e /var/run/openvpn ]; then
        mkdir -m 0770 /var/run/openvpn
        chown nobody:nogroup /var/run/openvpn
    fi
end script

exec /usr/sbin/openvpn --config /etc/openvpn/devops/config
post-stop exec sleep 5

不使用的替代脚本start-stop-daemon，在脚本节中发送SIGTERM到主进程pre-stop，可能如下所示：

我pre-stop按照您的建议添加了一个脚本：

start on (net-device-up
          and local-filesystems
          and runlevel [2345]
          and started rsyslog)
stop on runlevel [!2345]

respawn
respawn limit 6 60

env PIDFILE="/var/run/openvpn/devops.pid"

pre-start script
    if [ ! -e /var/run/openvpn ]; then
        mkdir -m 0770 /var/run/openvpn
        chown nobody:nogroup /var/run/openvpn
    fi
end script

exec /usr/sbin/openvpn --config /etc/openvpn/devops/config

pre-stop script
  PID=`cat $PIDFILE`
  kill -15 $PID
  sleep 1
  if [ "$?" -eq 0 ]; then
    rm -f $PIDFILE
  else
    echo "Unable to stop VPN '$NAME'"
  fi
end script

post-stop exec sleep 5

然后我尝试重新启动使用restart openvpn-devops但仍然出现此问题：

May 25 02:41:54 t-xhyve openvpn-devops-upstart: RTNETLINK answers: Operation not permitted
May 25 02:41:54 t-xhyve openvpn-devops-upstart: DOWN-ROOT: Error sending script execution signal to background process
May 25 02:41:54 t-xhyve openvpn-devops-upstart: DOWN-ROOT: BACKGROUND: INIT command='/bin/cat /proc/self/status > /tmp/temp.txt'

答案1

我尝试按相反的顺序回答这些问题：

2. 我为什么会得到这个信息？

DOWN-ROOT: Error sending script execution signal to background process

down-root使用该插件时，主进程将fork()在放弃 root 权限之前执行。然后，模块将保持等待状态，直到它通过管道从主进程收到执行 down-script 的消息（请参阅自述）。当主进程down-root收到时，会发送该消息给模块进程SIGTERM。

正如 OpenVPN 中提到的漏洞对于服务配置文件，此错误很可能与（ubuntu 14.04 使用的默认 init 守护程序）systemd一起导致，这将发送给两个进程。模块的进程将在调用 down-script 之前终止，然后才有机会完成其工作。upstartSIGTERMdown-root

使用/etc/init.d/openvpnUbuntu 附带的启动和停止服务或在upstart脚本中修复它。一个非常简单的工作 upstart 脚本可能如下所示。请注意，它用于start-stop-daemon发送正确的信号。

创建文件/etc/init/openvpn.conf

start on (net-device-up
          and local-filesystems
          and runlevel [2345]
          and started rsyslog)
stop on runlevel [!2345]

respawn
respawn limit 6 60

env PIDFILE="/var/run/openvpn/devops.pid"

pre-start script
    if [ ! -e /var/run/openvpn ]; then
        mkdir -m 0770 /var/run/openvpn
        chown nobody:nogroup /var/run/openvpn
    fi
end script

exec start-stop-daemon --start --quiet --oknodo \
  --pidfile /var/run/openvpn/devops.pid \
  --exec /usr/sbin/openvpn -- --config /etc/openvpn/devops/config < /dev/null

pre-stop script
  exec start-stop-daemon --stop --quiet --oknodo \
    --pidfile $PIDFILE --exec /usr/sbin/openvpn --retry 5
  if [ "$?" -eq 0 ]; then
    rm -f $PIDFILE
  else
    echo "Unable to stop VPN"
  fi
end script

post-stop exec sleep 5

一个选择不使用的脚本start-stop-daemon，在脚本节中发送SIGTERM到主进程pre-stop，可能如下所示：

start on (net-device-up
          and local-filesystems
          and runlevel [2345]
          and started rsyslog)
stop on runlevel [!2345]

respawn
respawn limit 6 60

env PIDFILE="/var/run/openvpn/devops.pid"

pre-start script
    if [ ! -e /var/run/openvpn ]; then
        mkdir -m 0770 /var/run/openvpn
        chown nobody:nogroup /var/run/openvpn
    fi
end script

exec /usr/sbin/openvpn --config /etc/openvpn/devops/config

pre-stop script
  PID=`cat $PIDFILE`
  kill -15 $PID
  sleep 1
  if [ "$?" -eq 0 ]; then
    rm -f $PIDFILE
  else
    echo "Unable to stop VPN"
  fi
end script

post-stop exec sleep 5

控制upstart工作：

initctl <command> openvpn
  • <command>：start，，stop​restart

这应该可以消除上述错误并down.sh正确运行脚本，但仍会出现以下错误：

Linux ip addr del failed: external program exited with error status: 2

首先出现这个错误，是因为ip addr del在TUN/TAP接口已经关闭后执行的。

下文将对此进行详细介绍。

1.OpenVPN为什么还会调用这个？

/sbin/ip addr del dev tun1 local 192.168.0.1 peer 192.168.0.2

这是由指令引起的ifconfig，它将在启动/停止服务时添加/删除 TUN/TAP 接口的 IP 地址。（参见手册页）

为了避免这种行为，请按如下方式修改配置。

OpenVPN 服务器配置文件：

topology net30
mode server
tls-server
push "route-gateway 192.168.0.1"
ifconfig 192.168.0.1 192.168.0.2
ifconfig-pool 192.168.0.5 192.168.0.9
ifconfig-noexec
up /usr/share/openvpn/up.sh
down-pre
plugin /usr/lib/openvpn/openvpn-plugin-down-root.so /usr/share/openvpn/down.sh
script-security 2
port 1196
proto udp
dev tun
keepalive 10 120
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status /var/log/openvpn/devops.log 1
syslog vpn-devops
verb 15
writepid /var/run/openvpn/devops.pid
ca /etc/openvpn/devops/ca.crt
cert /etc/openvpn/devops/server.crt
key /etc/openvpn/devops/server.key
dh /etc/openvpn/dh.pem
ifconfig-pool-persist /var/lib/openvpn/devops.ipp
client-config-dir /etc/openvpn/devops/ccd
crl-verify /etc/openvpn/devops/crl.pem
client-to-client
  • ifconfig-noexec：不执行ifconfig命令，将--ifconfig参数传递给脚本。
  • up：需要运行上行脚本来设置 IP 地址，原因是ifconfig-noexec
  • down-pre：--down在 TUN/TAP 关闭之前而不是之后调用脚本。
  • script-security 2：允许调用内置可执行文件和用户定义的脚本。

UP 脚本 /usr/share/openvpn/up.sh：

#!/bin/sh

/sbin/ip addr add dev $1 local $4 peer $5
/sbin/ip link set dev $1 up

DOWN 脚本 /usr/share/openvpn/down.sh：

#!/bin/sh

/sbin/ip addr del dev $1 local $4 peer $5
/sbin/ip link set dev $1 down

UP / DOWN 脚本也将启动或关闭接口（尽管不是必需的）。

确保它们是可执行的：

chmod +x /usr/share/openvpn/up.sh
chmod +x /usr/share/openvpn/down.sh

或者使用持久 TUN 接口

OpenVPN 服务器配置文件（编辑/添加这些行）：

dev tun1
# down-pre
  • dev tun1：我们将使用持久的 TUN 接口
  • # down-pre：不需要使用持久接口

最后创建一个持久 TUN 接口：

openvpn --mktun --dev tun1

这会构建一个持久隧道tun1，该隧道在 OpenVPN 的多个实例中有效，并且只有在被删除openvpn --rmtun --dev tun1或机器重新启动时才会失效。

故障排除

如果配置正确如上所示但仍然出现以下错误，请执行以下一些额外的故障排除步骤：

DOWN-ROOT: Error sending script execution signal to background process

启动 OpenVPN 实例，例如upstart：

initctl start openvpn

检查PID进程有什么：

ps -ef | grep "[o]penvpn"

# or more specific for our devops server example
ps -ef | grep "[o]penvpn/devops"

# this should show 2 openvpn processes
UID        PID  PPID  C STIME TTY          TIME CMD
nobody    3590     1  0 21:49 ?        00:00:00 /usr/sbin/openvpn --config /etc/openvpn/devops/config
root      3593  3590  0 21:49 ?        00:00:00 /usr/sbin/openvpn --config /etc/openvpn/devops/config

在跟踪原木时，杀nobody使用其SIGTERM​PID

kill -15 3590

或者或者一行即可：

kill -15 `ps -ef | grep "^[n]obody .* /usr/sbin/openvpn --config /etc/openvpn/devops/config$" | awk '{print $2}'`

压缩的日志输出看起来应如下所示（不带DOWN-ROOT: Error ...）：

vpn-devops[3590]: TCP/UDP: Closing socket
vpn-devops[3590]: PLUGIN_CALL: PRE type=PLUGIN_DOWN
upstart-devops  : DOWN-ROOT: BACKGROUND: received command code: 0
vpn-devops[3590]: PLUGIN_CALL: POST /usr/lib/openvpn/openvpn-plugin-down-root.so/PLUGIN_DOWN status=0
vpn-devops[3590]: Closing TUN/TAP interface
vpn-devops[3590]: PLUGIN_CLOSE: /usr/lib/openvpn/openvpn-plugin-down-root.so
vpn-devops[3590]: PID packet_id_free
vpn-devops[3590]: SIGTERM[hard,] received, process exiting
upstart-devops  : DOWN-ROOT: close
upstart-devops  : DOWN-ROOT: BACKGROUND: received command code: 1
upstart-devops  : DOWN-ROOT: BACKGROUND: EXIT
kernel: init    : ovpn-devops main process ended, respawning

相关内容