我是 Kubernetes 新手，在使用 Multus 插件时遇到一些问题。我使用的是 CentOS-7 和 minikube。
$ minikube version
minikube version: v1.18.0
commit: ec61815d60f66a6e4f6353030a40b12362557caa-dirty
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
我正在尝试将 multus 插件与 K8s 集成。为此，我从以下位置获取了 multus 的源代码： https://github.com/k8snetworkplumbingwg/multus-cni
使用 ./hack/build-go.sh 进行编译
然后我一直在关注https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/quickstart.md。
但是，pod 永远不会被创建；它永远停留在 ContainerCreating 状态。
我在 yml 文件中使用以下内容创建网络附件定义并执行 kubectl apply -f：
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "ens33",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.152.0/24",
"rangeStart": "192.168.152.200",
"rangeEnd": "192.168.152.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.152.1"
}
}'
这是 kubectl describe 的输出：
Name: macvlan-conf
Namespace: default
Labels: <none>
Annotations: <none>
API Version: k8s.cni.cncf.io/v1
Kind: NetworkAttachmentDefinition
Metadata:
Creation Timestamp: 2021-03-16T17:07:38Z
Generation: 1
Managed Fields:
API Version: k8s.cni.cncf.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:config:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-03-16T17:07:38Z
Resource Version: 102230
UID: 8473cdbf-1906-48b8-b180-e6f49502254d
Spec:
Config: { "cniVersion": "0.3.0", "type": "macvlan", "master": "ens33", "mode": "bridge", "ipam": { "type": "host-local", "subnet": "192.168.152.0/24", "rangeStart": "192.168.152.200", "rangeEnd": "192.168.152.216", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.152.1" } }
Events: <none>
然后我使用以下 yml 和 kubectl apply -f 创建 pod：
apiVersion: v1
kind: Pod
metadata:
name: samplepod1
spec:
containers:
- name: samplepod1
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
kubectl get pods 的输出是（samplepod1 是相关的）：
NAME READY STATUS RESTARTS AGE
samplepod 0/1 Terminating 0 54m
samplepod1 0/1 ContainerCreating 0 35s
kubectl describe pods 给我：
Namespace: default
Priority: 0
Node: minikube/192.168.49.2
Start Time: Tue, 16 Mar 2021 13:15:08 -0400
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
samplepod1:
Container ID:
Image: alpine
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/ash
-c
trap : TERM INT; sleep infinity & wait
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7vg94 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-7vg94:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-7vg94
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 64s default-scheduler Successfully assigned default/samplepod1 to minikube
Warning FailedCreatePodSandBox 62s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "26bcf14c4ec6cf8b532233abeefd7f6d9b2493b62963692d6460215dbb86916d" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 59s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e478b9559dc0837e2f1e1a78f1ca4f40b521a5176b9680ffefcaa4b485981a06" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 56s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7749375a5a4598313bace4a83d14e146af73e686098f8d82ccf1700d49663a1b" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 53s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "0d5c368e0b953c7463937d7c8f659b8554a467187a9f28462a348a7b64534689" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 50s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a93077c4f10db6c81d6ad23e353ebe024d32b5cbaf12e6618255c5286890ba34" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 47s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cc760d4034006f563bea6a4173e7fe0ae10e8bebe24327c9b28798121c9ca5d3" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 44s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "df255fbcd396c7f3bca09bce56220285aef8f063016ebc71f78428deb7ee24fb" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 41s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "62e3b999cb14cb116a2245d383fc05f819d0786da4c0ee4a24ce64fb239a9851" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Warning FailedCreatePodSandBox 39s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "07464566ca12539d54e73c3bf1ac73faebed6634703250954a8329e948954faf" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
Normal SandboxChanged 29s (x12 over 61s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 26s (x4 over 36s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2c75fc7558d368e364a3c48793deb93dd77b6579b9a47eb8a82a7b4037ae344d" network for pod "samplepod1": networkPlugin cni failed to set up pod "samplepod1_default" network: [default/samplepod1:crio]: error adding container to network "crio": failed to set bridge addr: could not add IP address to "cni0": permission denied
在这种情况下，我甚至没有在 pod 规范中使用“注释”。
有人可以帮我调试这个吗？
已编辑：我重新执行了整个过程。需要注意的是，我必须安装一个全新的 RHEL8 VM，因为我已经完全破坏了之前的 RHEL8 VM。现在的结果仍然是失败，但略有不同，如下所示。
遵循的步骤如下：
$ sudo systemctl start docker
$ minikube start --driver=docker
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane,master 24h v1.20.2
$ git clone https://github.com/intel/multus-cni.git && cd multus-cni
$ cat ./images/multus-daemonset.yml | kubectl apply -f -
此时，值得注意的是 /opt/cni/bin 目录中没有任何内容（甚至没有创建），/etc/cni/net.d/multus 目录中的情况也一样。
接下来是：
$ kubectl get pods --all-namespaces | grep multus
kube-system kube-multus-ds-khd6k 1/1 Running 0 3m7s
$ cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan-conf
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "ens160",
"mode": "bridge",
"ipam": {
"type": "host-local",
"subnet": "192.168.1.0/24",
"rangeStart": "192.168.1.200",
"rangeEnd": "192.168.1.216",
"routes": [
{ "dst": "0.0.0.0/0" }
],
"gateway": "192.168.1.1"
}
}'
EOF
networkattachmentdefinition.k8s.cni.cncf.io/macvlan-conf created
$ kubectl describe network-attachment-definitions macvlan-conf
Name: macvlan-conf
Namespace: default
Labels: <none>
Annotations: <none>
API Version: k8s.cni.cncf.io/v1
Kind: NetworkAttachmentDefinition
Metadata:
Creation Timestamp: 2021-03-17T18:41:34Z
Generation: 1
Managed Fields:
API Version: k8s.cni.cncf.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:config:
Manager: kubectl-create
Operation: Update
Time: 2021-03-17T18:41:34Z
Resource Version: 1597
UID: 29e54d08-70ba-4ad6-8031-f22ad2a73382
Spec:
Config: { "cniVersion": "0.3.0", "type": "macvlan", "master": "eth0", "mode": "bridge", "ipam": { "type": "host-local", "subnet": "192.168.1.0/24", "rangeStart": "192.168.1.200", "rangeEnd": "192.168.1.216", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.1.1" } }
Events: <none>
$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: samplepod
annotations:
k8s.v1.cni.cncf.io/networks: macvlan-conf
spec:
containers:
- name: samplepod
command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
image: alpine
EOF
pod/samplepod created
$ kubectl exec -it samplepod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:11:00:06 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.6/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
我还期望有一个“net1”接口，如网页上的描述所示： https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/quickstart.md
/opt 和 /etc 下的 cni 目录从未被创建。
提前感谢您的帮助。