我正在尝试在 Google Cloud 上安装以下平台:
2 个私有(本机)GKE 集群,位于 2 个不同的 VPC 中,为了让它们可以访问互联网,每个 vpc 都配置了一个 Cloud Nat。
我需要的是 2 个 GKE 集群进行交互,但在对等 VPC 时,我只能获得 POD 之间的通信,而不是 POD -> 服务或 POD -> 内部负载均衡器之间的通信。
集群:
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
Shrek01 asia-east1-a 1.16.8-gke.15 <none> g1-small 1.16.8-gke.15 3 RUNNING
Shrek02 asia-east2-a 1.15.9-gke.24 <none> g1-small 1.15.9-gke.24 3 RUNNING
vpcs:
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
Shrek01 CUSTOM REGIONAL
Shrek02 CUSTOM REGIONAL
子网:
NAME REGION NETWORK RANGE
Shrek01 asia-east1 Shrek01 192.168.13.0/24
Shrek02 asia-east2 Shrek02 192.168.14.0/24
对等:
NAME NETWORK PEER_PROJECT PEER_NETWORK AUTO_CREATE_ROUTES STATE STATE_DETAILS
Shrek01-Shrek01-peering Shrek01 pocprod2-2019001 Shrek02 True ACTIVE [2020-05-16T14:29:57.864-07:00]: Connected.
Shrek02-Shrek01-peering Shrek02 pocprod2-2019001 Shrek01 True ACTIVE [2020-05-16T14:29:57.864-07:00]: Connected.
防火墙规则:
- “Shrek01-peering-ingress”
{
"allowed": [
{
"IPProtocol": "all"
}
],
"creationTimestamp": "2020-05-16T16:05:14.829-07:00",
"description": "",
"direction": "INGRESS",
"disabled": false,
"id": "6807007164648771397",
"kind": "compute#firewall",
"logConfig": {
"enable": false
},
"name": "peering-ingress",
"network": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/networks/Shrek01",
"priority": 1000,
"selfLink": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/firewalls/peering-ingress",
"sourceRanges": [
"192.168.14.0/24",
"10.113.64.0/19",
"10.213.64.0/19"
]
}
- “Shrek02-peering-ingress”
{
"allowed": [
{
"IPProtocol": "all"
}
],
"creationTimestamp": "2020-05-16T16:24:28.545-07:00",
"description": "",
"direction": "INGRESS",
"disabled": false,
"id": "7130188648920500419",
"kind": "compute#firewall",
"logConfig": {
"enable": false
},
"name": "Shrek02-peering-ingress",
"network": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/networks/Shrek02",
"priority": 1000,
"selfLink": "https://www.googleapis.com/compute/v1/projects/pocprod2-2019001/global/firewalls/Shrek02-peering-ingress",
"sourceRanges": [
"192.168.13.0/24",
"10.113.32.0/19",
"10.213.32.0/19"
]
}
k8s Shrek01集群:
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.213.32.1 <none> 443/TCP 85m <none>
nginx LoadBalancer 10.213.60.14 192.168.13.7 80:32612/TCP 92s app=nginx
nginx-cip ClusterIP 10.213.34.24 <none> 80/TCP 93s app=nginx
nginx-np NodePort 10.213.35.31 <none> 80:30444/TCP 92s app=nginx
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-64b4f9bb85-9sjcp 1/1 Running 0 3m34s 10.113.34.11 gke-Shrek01-default-pool-f9ecbfcc-dz9z <none> <none>
nginx-64b4f9bb85-l2bzd 1/1 Running 0 3m34s 10.113.32.5 gke-Shrek01-default-pool-f9ecbfcc-pdll <none> <none>
nginx-64b4f9bb85-xd7kw 1/1 Running 0 3m34s 10.113.33.9 gke-Shrek01-default-pool-f9ecbfcc-v67d <none> <none>
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
gke-Shrek01-default-pool-f9ecbfcc-dz9z Ready <none> 89m v1.16.8-gke.15 192.168.13.4 Container-Optimized OS from Google 4.19.109+ docker://19.3.1
gke-Shrek01-default-pool-f9ecbfcc-pdll Ready <none> 89m v1.16.8-gke.15 192.168.13.2 Container-Optimized OS from Google 4.19.109+ docker://19.3.1
gke-Shrek01-default-pool-f9ecbfcc-v67d Ready <none> 89m v1.16.8-gke.15 192.168.13.3 Container-Optimized OS from Google 4.19.109+ docker://19.3.1
- 来自 Shrek02 吊舱的检查:
root@nginx-5c66c56f55-8jwv2:/# echo ${MY_POD_IP}
10.113.66.9
# internal load balancer
root@nginx-5c66c56f55-8jwv2:/# nc -vz 192.168.13.7 80
192.168.13.7: inverse host lookup failed: Unknown host
(UNKNOWN) [192.168.13.7] 80 (?) : Connection timed out
# intarnal load balancer's Cluster IP
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.213.60.14 80
10.213.60.14: inverse host lookup failed: Unknown host
(UNKNOWN) [10.213.60.14] 80 (?) : Connection timed out
# ClusterIP
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.213.34.24 80
10.213.34.24: inverse host lookup failed: Unknown host
(UNKNOWN) [10.213.34.24] 80 (?) : Connection timed out
# NodePort
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.213.35.31 80
10.213.35.31: inverse host lookup failed: Unknown host
(UNKNOWN) [10.213.35.31] 80 (?) : Connection timed out
# Pod IP
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.113.34.11 80
10.113.34.11: inverse host lookup failed: Unknown host
(UNKNOWN) [10.113.34.11] 80 (?) open
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.113.32.5 80
10.113.32.5: inverse host lookup failed: Unknown host
(UNKNOWN) [10.113.32.5] 80 (?) open
root@nginx-5c66c56f55-8jwv2:/# nc -vz 10.113.33.9 80
10.113.33.9: inverse host lookup failed: Unknown host
(UNKNOWN) [10.113.33.9] 80 (?) open
我是不是忘记了什么步骤?我没有发现错误。
答案1
- 我能够使用类似的环境将 pod 连接到 pod、将 pod 连接到节点端口以及将端口连接到外部负载均衡器,但需要注意的是:
内部负载均衡器仅在以下情况下才允许在 VPC Peering 上使用:
- 对等网络中的客户端虚拟机 (VM) 实例与内部负载均衡器位于同一区域
- 您配置全球访问。配置全局访问后,对等 VPC 网络的任何区域的客户端 VM 实例都可以访问您的内部 TCP/UDP 负载均衡器。内部 HTTP(S) 负载均衡不支持全局访问。
为了在不同区域使用内部全球访问和 VPC 对等连接,您需要两个选择:
- 如果你知道内部负载均衡器名称,可以使用以下命令:
$ gcloud compute forwarding-rules update <LB_NAME> \
--region=<REGION> \
--allow-global-access
- 并使用以下命令进行检查:
gcloud compute forwarding-rules describe <LB_NAME> \
--region=us-west1 \
--format="get(name,region,allowGlobalAccess)"
- 如果你所在地区有少量内部负载均衡,另一种简单的更改方法是通过 GCP 负载均衡页面单击此处访问:
- 在里面姓名列中,单击集群区域的内部 TCP/UDP 负载均衡器(单击后,您将看到子网名称,如下例所示):
- 然后点击编辑。
- 点击前端配置
- 点击铅笔编辑
- 在下面全球访问, 选择 使能够。
- 点击完毕
- 打开并点击更新更新规则。
- 等到规则应用完成。
- 改变之后,我就可以从《Shrek02》中瞄准《Shrek01》中的内部 LB。(我将在下面给出例子)。
笔记:
ClusterIP
:在集群内部 IP 上公开服务。选择此值会使服务仅可从集群内部访问。因此您将不会被路由到外部访问。
再生产:
- 我按照您的参数创建了两个 VPC:
$ gcloud container clusters list
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
shrek01 europe-west1-b 1.16.8-gke.15 XX.XXX.XX.XXX g1-small 1.16.8-gke.15 3 RUNNING
shrek02 europe-west2-b 1.15.9-gke.24 XXX.XXX.XX.XXX g1-small 1.15.9-gke.24 3 RUNNING
$ gcloud compute networks subnets list
NAME REGION NETWORK RANGE
shrek01 europe-west1 shrek01 192.168.13.0/24
shrek02 europe-west2 shrek02 192.168.14.0/24
$ gcloud compute networks peerings list-routes sh1-sh2 --network=shrek01 --region europe-west1 --direction=INCOMING
DEST_RANGE TYPE NEXT_HOP_REGION PRIORITY STATUS
192.168.14.0/24 SUBNET_PEERING_ROUTE europe-west2 1000 accepted
10.229.0.0/20 SUBNET_PEERING_ROUTE europe-west2 1000 accepted
10.36.0.0/14 SUBNET_PEERING_ROUTE europe-west2 1000 accepted
$ gcloud compute networks peerings list-routes sh2-sh1 --network=shrek02 --region europe-west2 --direction=INCOMING
DEST_RANGE TYPE NEXT_HOP_REGION PRIORITY STATUS
192.168.13.0/24 SUBNET_PEERING_ROUTE europe-west1 1000 accepted
10.154.0.0/20 SUBNET_PEERING_ROUTE europe-west1 1000 accepted
10.24.0.0/14 SUBNET_PEERING_ROUTE europe-west1 1000 accepted
确保我的节点可以在 VPC 之间进行 Ping 后,我将使用此 yaml 测试入口和连接
hello-1.yaml
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-1
spec:
replicas: 3
selector:
matchLabels:
app: hello-1
template:
metadata:
labels:
app: hello-1
spec:
containers:
- name: hello-1
image: gcr.io/google-samples/hello-app:1.0
ports:
- name: http
containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: hello-1-svc
spec:
type: NodePort
selector:
app: hello-1
ports:
- protocol: TCP
port: 80
targetPort: 8080
hello-2.yaml
:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-2
spec:
replicas: 3
selector:
matchLabels:
app: hello-2
template:
metadata:
labels:
app: hello-2
spec:
containers:
- name: hello-2
image: gcr.io/google-samples/hello-app:2.0
ports:
- name: http
containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: hello-2-svc
spec:
type: NodePort
selector:
app: hello-2
ports:
- protocol: TCP
port: 80
targetPort: 8080
hello-ingress.yaml
:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: hello-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host:
http:
paths:
- path: /
backend:
serviceName: hello-1-svc
servicePort: 80
- path: /v2
backend:
serviceName: hello-2-svc
servicePort: 80
- 查看 pod 名称和 pod 的 IP、以及节点 IP 和负载均衡器/nodeport 端口。
o$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-1-84d5994678-dx8dv 1/1 Running 0 140m 10.24.0.9 gke-shrek01-default-pool-5ffc38d7-bz35 <none> <none>
hello-1-84d5994678-t74mn 1/1 Running 0 14m 10.24.1.3 gke-shrek01-default-pool-5ffc38d7-70sk <none> <none>
hello-1-84d5994678-zq7t2 1/1 Running 0 14m 10.24.2.9 gke-shrek01-default-pool-5ffc38d7-zfj6 <none> <none>
hello-2-5c4f554ccc-b8j6f 1/1 Running 0 140m 10.24.0.10 gke-shrek01-default-pool-5ffc38d7-bz35 <none> <none>
hello-2-5c4f554ccc-km4ph 1/1 Running 0 13m 10.24.1.4 gke-shrek01-default-pool-5ffc38d7-70sk <none> <none>
hello-2-5c4f554ccc-z4f6n 1/1 Running 0 13m 10.24.2.10 gke-shrek01-default-pool-5ffc38d7-zfj6 <none> <none>
$ ubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-1-svc NodePort 10.154.13.186 <none> 80:32186/TCP 140m
hello-2-svc NodePort 10.154.4.214 <none> 80:32450/TCP 140m
$ kubectl get svc ingress-nginx-controller -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.154.10.104 192.168.13.5 80:30112/TCP,443:32156/TCP 4h20m
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
hello-ingress * 192.168.13.5 80 98m
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
gke-shrek01-default-pool-5ffc38d7-70sk Ready <none> 2d19h v1.16.8-gke.15 192.168.13.3 XX.XXX.XX.XXX Container-Optimized OS from Google 4.19.109+ docker://19.3.1
gke-shrek01-default-pool-5ffc38d7-bz35 Ready <none> 2d19h v1.16.8-gke.15 192.168.13.2 XXX.XXX.XX.XXX Container-Optimized OS from Google 4.19.109+ docker://19.3.1
gke-shrek01-default-pool-5ffc38d7-zfj6 Ready <none> 2d19h v1.16.8-gke.15 192.168.13.4 XX.XXX.X.XXX Container-Optimized OS from Google 4.19.109+ docker://19.3.1
现在我将连接到shrek02
集群,创建一个 pod 并安装curl
:
project@cloudshell:~$ kubectl run ubuntu --image=ubuntu -it -- /bin/bash
root@ubuntu:/# apt update
root@ubuntu:/# apt install curl
root@ubuntu:/# exit
project@cloudshell:~$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ubuntu 1/1 Running 1 2m51s 10.36.1.6 gke-shrek02-default-pool-a7a08ac8-0lrz <none> <none>
- 您可以看到我们位于
shrek02
,现在让我们测试与资源的连接shrek01
。请记住kube-dns
仅在集群内部可用,因此我们将使用 IP 进行连接:
project@cloudshell:~$ kubectl exec -it ubuntu -- /bin/bash
###Hello-1 POD:
root@ubuntu:/# curl 10.24.0.9:8080
Hello, world!
Version: 1.0.0
Hostname: hello-1-84d5994678-dx8dv
###Hello-2 POD:
root@ubuntu:/# curl 10.24.1.4:8080
Hello, world!
Version: 2.0.0
Hostname: hello-2-5c4f554ccc-km4ph
### HELLO-1-SVC USING NODE IP + NODEPORT:
root@ubuntu:/# curl 192.168.13.3:32186
Hello, world!
Version: 1.0.0
Hostname: hello-1-84d5994678-t74mn
### HELLO-2-SVC USING ANOTHER NODE IP + NODEPORT:
root@ubuntu:/# curl 192.168.13.2:32450
Hello, world!
Version: 2.0.0
Hostname: hello-2-5c4f554ccc-km4ph
### NOW LET'S TEST OUR INGRESS which routes "/" to hello-1 and "/v2" to hello-2:
root@ubuntu:/# curl 192.168.13.5/
Hello, world!
Version: 1.0.0
Hostname: hello-1-84d5994678-dx8dv
root@ubuntu:/# curl 192.168.13.5/v2
Hello, world!
Version: 2.0.0
Hostname: hello-2-5c4f554ccc-b8j6f
我希望它能帮助您解决您的环境问题,如果您有任何问题,请在评论中告诉我。