☰

来自 upstart 的嵌套非特权 lxc 容器，其所有者可以停止

tag-icon • tag-icon 14.04 lxc cgroup unprivileged
来自 upstart 的嵌套非特权 lxc 容器，其所有者可以停止

在运行Ubuntu 14.04.5 LTS我有一个名为西可以创建一个启动非特权 lxc 容器，同时运行Ubuntu 14.04.5 LTS. 该用户的 subid 范围200000-231071。这样的容器的配置文件为：

# Distribution configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.include = /usr/share/lxc/config/ubuntu.userns.conf
lxc.arch = x86_64

# Nested
lxc.mount.auto = cgroup
lxc.aa_profile = lxc-container-default-with-nesting

# Container specific configuration
lxc.id_map = u 0 200000 65536
lxc.id_map = u 100000 265536 65536
lxc.id_map = g 0 200000 65536
lxc.id_map = g 100000 265536 65536
lxc.rootfs = /home/ci/.local/share/lxc/ci/rootfs
lxc.utsname = ci

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.hwaddr = 00:16:3e:dd:f1:99

用户可以毫无问题地创建并启动非特权容器：

ci@host:~$ lxc-create -t download -n ci -- -d ubuntu -r trusty -a amd64
ci@host:~$ lxc-start -n ci -d
ci@host:~$ lxc-ls --fancy
    NAME  STATE    IPV4                 IPV6  AUTOSTART
    ---------------------------------------------------
    ci    RUNNING  10.0.3.75, 10.0.4.1  -     NO

在主机中，管理员在跑：

root@host ~ # ps ax | grep cgmanager
    382 ?        Ss     0:01 /sbin/cgmanager --sigstop -m name=systemd

在非特权容器中西，代理服务器在跑：

root@ci:~# ps ax | grep cgproxy
    288 ?        Ss     0:00 /sbin/cgproxy --sigstop

在非特权容器中西，名为詹金斯具有 subid 范围100000-65535可以在其中创建并启动非特权容器，即非特权嵌套容器，不过也有一些技巧，具体如下：

  1. 使用以下方式登录后远程控制作为用户詹金斯在非特权容器中西，其结果cat /proc/self/cgroup为：

    jenkins@ci:~$ cat /proc/self/cgroup
    12:hugetlb:/user/1012.user/11.session/lxc/ci
    11:net_prio:/user/1012.user/11.session/lxc/ci
    10:perf_event:/user/1012.user/11.session/lxc/ci
    9:net_cls:/user/1012.user/11.session/lxc/ci
    8:freezer:/user/1012.user/11.session/lxc/ci
    7:devices:/user/1012.user/11.session/lxc/ci
    6:memory:/user/1012.user/11.session/lxc/ci
    5:blkio:/user/1012.user/11.session/lxc/ci
    4:name=systemd:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    3:cpuacct:/user/1012.user/11.session/lxc/ci
    2:cpu:/user/1012.user/11.session/lxc/ci
    1:cpuset:/user/1012.user/11.session/lxc/ci

  2. 在此刻，詹金斯可以创建容器，但无法启动它：

    jenkins@ci:~$ lxc-create -t download -n test -- -d ubuntu -r trusty -a amd64
jenkins@ci:~$ lxc-start -n test
    lxc_container: cgmanager.c: lxc_cgmanager_create: 301 call to cgmanager_create_sync failed: invalid request
    lxc_container: cgmanager.c: lxc_cgmanager_create: 303 Failed to create hugetlb:lxc/test
    lxc_container: cgmanager.c: cgm_create: 650 Error creating cgroup hugetlb:lxc/test
    lxc_container: start.c: lxc_spawn: 891 failed creating cgroups
    lxc_container: start.c: __lxc_start: 1121 failed to spawn 'test'
    lxc_container: lxc_start.c: main: 341 The container failed to start.
    lxc_container: lxc_start.c: main: 345 Additional information can be obtained by setting the --logfile and --logpriority options.

  3. 我以 root 身份在容器中发出：

    restart systemd-logind

  4. 现在作为用户詹金斯在容器中，我注销并再次登录远程控制。cgroup 已经改变，现在我可以创建并运行一个容器：

    jenkins@ci:~$ cat /proc/self/cgroup
    12:hugetlb:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    11:net_prio:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    10:perf_event:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    9:net_cls:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    8:freezer:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    7:devices:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    6:memory:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    5:blkio:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    4:name=systemd:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    3:cpuacct:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    2:cpu:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
    1:cpuset:/user/1012.user/11.session/lxc/ci/user/1012.user/11.session/lxc/ci/user/107.user/c1.session
jenkins@ci:~$ lxc-create -t download -n test -- -d ubuntu -r trusty -a amd64
jenkins@ci:~$ lxc-start -n test -d
jenkins@ci:~$ lxc-ls --fancy
    NAME     STATE    IPV4       IPV6  AUTOSTART
    --------------------------------------------
    test     RUNNING  10.0.4.64  -     NO

第一个问题：为什么我需要这样做restart systemd-logind，以及如何避免在能够创建嵌套的非特权容器之前以 root 身份输入它？

在容器中西我已经创建了一个 init 配置文件（upstart conf 文件位于/etc/init/jenkins.conf）运行软件詹金斯作为用户詹金斯：

description "jenkins"

start on filesystem and static-network-up
stop on runlevel [016]

env USER="jenkins"
env GROUP="jenkins"
env HOME="/var/lib/jenkins"
env JENKINS_LOG="/var/log/jenkins"
env JENKINS_ROOT="/usr/share/jenkins"
env JENKINS_RUN="/var/run/jenkins"
env JENKINS_PIDFILE="jenkins.pid"

pre-start script
    test -f $JENKINS_ROOT/jenkins.war || { stop ; exit 0; }
    mkdir $JENKINS_RUN > /dev/null 2>&1  || true
    chown -R $USER:$GROUP $JENKINS_RUN || true
    mkdir $JENKINS_LOG > /dev/null 2>&1  || true
    chown -R $USER:$GROUP $JENKINS_LOG || true
end script

script
    . /etc/default/jenkins
    # export XDG_SESSION_ID="/run/user/`id -u $USER`"
    export HOME
    export USER
    export GROUP
    exec daemon --name=jenkins --foreground --inherit --user=$USER:$GROUP --pidfile=$JENKINS_RUN/$JENKINS_PIDFILE --output=$JENKINS_LOG -- $JAVA $JAVA_ARGS -jar $JENKINS_WAR $JENKINS_ARGS
end script

post-start script
    while [ ! -f $JENKINS_RUN/$JENKINS_PIDFILE ]; do sleep 1; done
    PID=$(cat $JENKINS_RUN/$JENKINS_PIDFILE)
    cgm create all $USER
    cgm chown all $USER $(id -u $USER) $(id -g $USER)
    # this need to be run in the jenkins job script:
    # cgm movepid all $USER $$
end script

# vim: ft=upstart

在脚本中，该过程詹金斯开始所谓的Jenkins 的构建，如果我添加以下行：

cgm movepid all $USER $$

该脚本可以创建并启动非特权嵌套容器，即其 cgroup：

+ cat /proc/self/cgroup
12:hugetlb:/user/1012.user/11.session/lxc/ci/jenkins
11:net_prio:/user/1012.user/11.session/lxc/ci/jenkins
10:perf_event:/user/1012.user/11.session/lxc/ci/jenkins
9:net_cls:/user/1012.user/11.session/lxc/ci/jenkins
8:freezer:/user/1012.user/11.session/lxc/ci/jenkins
7:devices:/user/1012.user/11.session/lxc/ci/jenkins
6:memory:/user/1012.user/11.session/lxc/ci/jenkins
5:blkio:/user/1012.user/11.session/lxc/ci/jenkins
4:name=systemd:/user/1012.user/11.session/lxc/ci/jenkins
3:cpuacct:/user/1012.user/11.session/lxc/ci/jenkins
2:cpu:/user/1012.user/11.session/lxc/ci/jenkins
1:cpuset:/user/1012.user/11.session/lxc/ci/jenkins

但一个用户詹金斯使用 ssh 登录无法停止脚本创建的容器。以下内容永远不会完成：

jenkins@ci:~$ lxc-stop -n test

第二个问题：我怎样才能让用户詹金斯可以停止用户创建的任何容器詹金斯从像上面的初始化脚本？

相关内容