☰

rabbitmq kubernetes 带有 NFS 挂载

tag-icon • tag-icon nfs kubernetes rabbitmq chown
rabbitmq kubernetes 带有 NFS 挂载

我尝试在具有 NFS PV 的 kubernetes 环境中设置 rabbitmq 集群，借助本教程。不幸的是，rabbitmq 似乎想要更改的所有者/usr/lib/rabbitmq，但是当我在那里安装 NFS 目录时，出现错误：

 $ kubectl logs rabbitmq-0 -f
chown: /var/lib/rabbitmq: Operation not permitted
chown: /var/lib/rabbitmq: Operation not permitted

我想我有两个选择：分叉rabbitmq并删除chown并构建我自己的图像或使 kubernetes/nfs 正常工作。我不想制作自己的 fork，让 kubernetes/nfs 正常工作听起来不应该是我的问题。还有其他想法吗？

答案1

这是我尝试重现此问题的方法。我已经kubeadm在 redhat 7 上安装了 kubernetes 集群，下面是集群、节点详细信息。

环境详情：

[root@master tmp]# kubectl cluster-info
Kubernetes master is running at https://192.168.56.4:6443
KubeDNS is running at https://192.168.56.4:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

要进一步调试和诊断集群问题，请使用kubectl cluster-info dump。

[根@主 tmp]

[root@master tmp]# kubectl get no
NAME         STATUS     ROLES    AGE     VERSION
master.k8s   Ready      master   8d      v1.16.2
node1.k8s    Ready   <none>   7d22h   v1.16.3
node2.k8s    Ready      <none>   7d21h   v1.16.3
[root@master tmp]#

nfs首先，我通过在主节点和工作节点上运行以下步骤来设置主节点和工作节点上的配置。这里主节点是nfs服务器，两个工作节点都是nfs客户端。

NFS 设置：

yum install nfs-utils nfs-utils-lib =============================================================>>>>> on nfs server,client
yum install portmap       =============================================================>>>>> on nfs server,client
mkdir /nfsroot =============================>>>>>>>>>>>>>>>>>>on nfs server
[root@master ~]# cat /etc/exports   =============================================================>>>>> on nfs server
/nfsroot 192.168.56.5/255.255.255.0(rw,sync,no_root_squash)
/nfsroot 192.168.56.6/255.255.255.0(rw,sync,no_root_squash)
exportfs -r               =============================================================>>>>> on nfs server
service nfs start =============================================================>>>>> on nfs server,client
showmount -e =============================================================>>>>> on nfs server,client

现在nfs设置已准备就绪并将应用rabbitmqk8s 设置。

RabbitMQ K8S 设置：

第一步是persistent volumes使用nfs mount我们在上一步中创建的。

[root@master tmp]# cat /root/rabbitmq-pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
 name: rabbitmq-pv-1
spec:
 accessModes:
 - ReadWriteOnce
 - ReadOnlyMany
 nfs:
  server: 192.168.56.4
  path: /nfsroot
 capacity:
  storage: 1Mi
 persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolume
apiVersion: v1
metadata:
 name: rabbitmq-pv-2
spec:
 accessModes:
 - ReadWriteOnce
 - ReadOnlyMany
 nfs:
  server: 192.168.56.4
  path: /nfsroot
 capacity:
  storage: 1Mi
 persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolume
apiVersion: v1
metadata:
 name: rabbitmq-pv-3
spec:
 accessModes:
 - ReadWriteOnce
 - ReadOnlyMany
 nfs:
  server: 192.168.56.4
  path: /nfsroot
 capacity:
  storage: 1Mi
 persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolume
apiVersion: v1
metadata:
 name: rabbitmq-pv-4
spec:
 accessModes:
 - ReadWriteOnce
 - ReadOnlyMany
 nfs:
  server: 192.168.56.4
  path: /nfsroot
 capacity:
  storage: 1Mi
 persistentVolumeReclaimPolicy: Recycle

应用上述清单后，它创建pv's如下内容：

[root@master ~]# kubectl apply -f rabbitmq-pv.yaml
persistentvolume/rabbitmq-pv-1 created
persistentvolume/rabbitmq-pv-2 created
persistentvolume/rabbitmq-pv-3 created
persistentvolume/rabbitmq-pv-4 created
[root@master ~]# kubectl get pv
NAME            CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
rabbitmq-pv-1   1Mi        RWO,ROX        Recycle          Available                                   5s
rabbitmq-pv-2   1Mi        RWO,ROX        Recycle          Available                                   5s
rabbitmq-pv-3   1Mi        RWO,ROX        Recycle          Available                                   5s
rabbitmq-pv-4   1Mi        RWO,ROX        Recycle          Available                                   5s
[root@master ~]#

无需创建，因为在运行清单时将通过选项PersistentVolumeClaim自动处理。现在让我们创建您提到的内容，如下所示：statefulsetvolumeclaimtemplatesecret

[root@master tmp]# kubectl create secret generic rabbitmq-config --from-literal=erlang-cookie=c-is-for-cookie-thats-good-enough-for-me
secret/rabbitmq-config created
[root@master tmp]#

[root@master tmp]# kubectl get secrets
NAME                  TYPE                                  DATA   AGE
default-token-vjsmd   kubernetes.io/service-account-token   3      8d
jp-token-cfdzx        kubernetes.io/service-account-token   3      5d2h
rabbitmq-config       Opaque                                1      39m
[root@master tmp]#

现在让我们通过将所有服务类型rabbitmq替换为服务来提交您的清单，因为我们没有使用任何云提供商环境。还将替换为我们在 pv 步骤中创建的 rabbitmq-pv。将大小从 1Gi 减少到 1Mi，因为这只是测试演示。loadbalancernodeportvolume names

apiVersion: v1
kind: Service
metadata:
  # Expose the management HTTP port on each node
  name: rabbitmq-management
  labels:
    app: rabbitmq
spec:
  ports:
  - port: 15672
    name: http
  selector:
    app: rabbitmq
  sessionAffinity: ClientIP
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  # The required headless service for StatefulSets
  name: rabbitmq
  labels:
    app: rabbitmq
spec:
  ports:
  - port: 5672
    name: amqp
  - port: 4369
    name: epmd
  - port: 25672
    name: rabbitmq-dist
  clusterIP: None
  selector:
    app: rabbitmq
---
apiVersion: v1
kind: Service
metadata:
  # The required headless service for StatefulSets
  name: rabbitmq-cluster
  labels:
    app: rabbitmq
spec:
  ports:
  - port: 5672
    name: amqp
  - port: 4369
    name: epmd
  - port: 25672
    name: rabbitmq-dist
  type: NodePort
  selector:
    app: rabbitmq
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: "rabbitmq"
  selector:
   matchLabels:
    app: rabbitmq
  replicas: 4
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: rabbitmq
        image: rabbitmq:3.6.6-management-alpine
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/sh
              - -c
              - >
                if [ -z "$(grep rabbitmq /etc/resolv.conf)" ]; then
                  sed "s/^search \([^ ]\+\)/search rabbitmq.\1 \1/" /etc/resolv.conf > /etc/resolv.conf.new;
                  cat /etc/resolv.conf.new > /etc/resolv.conf;
                  rm /etc/resolv.conf.new;
                fi;
                until rabbitmqctl node_health_check; do sleep 1; done;
                if [[ "$HOSTNAME" != "rabbitmq-0" && -z "$(rabbitmqctl cluster_status | grep rabbitmq-0)" ]]; then
                  rabbitmqctl stop_app;
                  rabbitmqctl join_cluster rabbit@rabbitmq-0;
                  rabbitmqctl start_app;
                fi;
                rabbitmqctl set_policy ha-all "." '{"ha-mode":"exactly","ha-params":3,"ha-sync-mode":"automatic"}'
        env:
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: rabbitmq-config
              key: erlang-cookie
        ports:
        - containerPort: 5672
          name: amqp
        - containerPort: 25672
          name: rabbitmq-dist
        volumeMounts:
        - name: rabbitmq-pv
          mountPath: /var/lib/rabbitmq
  volumeClaimTemplates:
  - metadata:
      name: rabbitmq-pv
      annotations:
        volume.alpha.kubernetes.io/storage-class: default
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Mi # make this bigger in production

提交清单后，我们就可以看到statefulset并创建了 Pod。

[root@master tmp]# kubectl apply -f rabbitmq.yaml
service/rabbitmq-management created
service/rabbitmq created
service/rabbitmq-cluster created
statefulset.apps/rabbitmq created

[root@master tmp]#
NAME                         READY   STATUS                       RESTARTS   AGE
rabbitmq-0                   1/1     Running                      0          18m
rabbitmq-1                   1/1     Running                      0          17m
rabbitmq-2                   1/1     Running                      0          13m
rabbitmq-3                   1/1     Running                      0          13m

[root@master ~]# kubectl get pvc
NAME                     STATUS   VOLUME          CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rabbitmq-pv-rabbitmq-0   Bound    rabbitmq-pv-1   1Mi        RWO,ROX                       49m
rabbitmq-pv-rabbitmq-1   Bound    rabbitmq-pv-3   1Mi        RWO,ROX                       48m
rabbitmq-pv-rabbitmq-2   Bound    rabbitmq-pv-2   1Mi        RWO,ROX                       44m
rabbitmq-pv-rabbitmq-3   Bound    rabbitmq-pv-4   1Mi        RWO,ROX                       43m

[root@master ~]# kubectl get svc
NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                         AGE
rabbitmq              ClusterIP   None             <none>        5672/TCP,4369/TCP,25672/TCP                     49m
rabbitmq-cluster      NodePort    10.102.250.172   <none>        5672:30574/TCP,4369:31757/TCP,25672:31854/TCP   49m
rabbitmq-management   NodePort    10.108.131.46    <none>        15672:31716/TCP                                 49m
[root@master ~]#

现在我尝试使用服务进入rabbitmq管理页面nodeporthttp://192.168.56.6://31716并且我能够获得登录页面rabbitmq管理。

登录页面

集群状态

因此，如果您尝试上述方法后仍然遇到 chown 问题，请告诉我，以便我们通过检查podsecuritypolicies是否应用来进一步了解。

相关内容