vSphere with Tanzu and Multi Zones part2
Overview
This post will explore a bit further how we can utilize the three-zone setup configured here for TKG cluster and application placement. It will not be a very long post, but showing how it can be done and which values to use. In the "part 1" post I enabled a three-zone Supervisor deployment, meaning the three Supervisor Control Plane VMs will be distributed evenly across my three vSphere Clusters. To use a three-zone deployment we need to have three vSphere Zones defined. Each of these vSphere zones is described as a "Failure Domain" and becomes a value we can use when we decide the placement of both TKG clusters and applications inside our TKG clusters. So basically this post will describe how I can take advantage of this when I deploy my workload clusters and how can I decide where my applications will be placed.
Workload cluster placement
My Supervisor cluster is already deployed in my three vSphere Zones, just waiting for me to give it something to do. I have created a vSphere Namespace for my TKG cluster called ns-three-zone-1. I want to use a different workload network than my Supervisor workload network is placed on, that is a benefit when using NSX-T.
To give some context, this is how my environment is looking before deploying the TKG cluster:
Now I just need to log into the supervisor, prepare my TKG yaml manifest and deploy my TKG cluster.
Log in to Supervisor
1linuxvm01:~/$ kubectl vsphere login --server=10.101.90.2 --insecure-skip-tls-verify --vsphere-username=andreasm@cpod-nsxam-wdc.az-wdc.cloud-garage.net
2
3
4KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
5Password:
6Logged in successfully.
7
8You have access to the following contexts:
9 10.101.90.2
10
11If the context you wish to use is not in this list, you may need to try
12logging in again later, or contact your cluster administrator.
13
14To change context, use `kubectl config use-context <workload name>`
15linuxvm01:~/$
Now that I am logged in, I will check if my vSphere Zones are available by issuing the following command:
1linuxvm01:~/three-zones$ k get vspherezones.topology.tanzu.vmware.com
2NAME AGE
3wdc-zone-1 19d
4wdc-zone-2 19d
5wdc-zone-3 19d
6linuxvm01:~/three-zones$
So it seems, now I need to use these names/labels in my TKG yaml manifest. In my first deployment I will use the example from the official VMware documentation here with some additions from my side like run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu I will be using the API v1beta1 with kubectl not Tanzu CLI.
Let us have a look at it and edit it accordingly:
1apiVersion: cluster.x-k8s.io/v1beta1
2kind: Cluster
3metadata:
4 name: three-zone-cluster-1 #My own name on the cluster
5 namespace: ns-three-zone-1 #My vSphere Namespace
6spec:
7 clusterNetwork:
8 services:
9 cidrBlocks: ["20.30.0.0/16"] #Edited by me
10 pods:
11 cidrBlocks: ["20.40.0.0/16"] #Edited by me
12 serviceDomain: "cluster.local"
13 topology:
14 class: tanzukubernetescluster
15 version: v1.24.9+vmware.1-tkg.4 #My latest available TKR version
16 controlPlane:
17 replicas: 1 # only one controlplane (saving resources and time)
18 metadata:
19 annotations:
20 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
21 workers:
22 #muliple node pools are used
23 machineDeployments:
24 - class: node-pool
25 name: node-pool-1
26 replicas: 1 #only 1 worker here
27 metadata:
28 annotations:
29 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
30 #failure domain the machines will be created in
31 #maps to a vSphere Zone; name must match exactly
32 failureDomain: wdc-zone-1 #named after my vSphere zone
33 - class: node-pool
34 name: node-pool-2
35 replicas: 1 #only 1 worker here
36 metadata:
37 annotations:
38 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
39 #failure domain the machines will be created in
40 #maps to a vSphere Zone; name must match exactly
41 failureDomain: wdc-zone-2 #named after my vSphere zone
42 - class: node-pool
43 name: node-pool-3
44 replicas: 1 #only 1 worker here
45 metadata:
46 annotations:
47 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
48 #failure domain the machines will be created in
49 #maps to a vSphere Zone; name must match exactly
50 failureDomain: wdc-zone-3 #named after my vSphere zone
51 variables:
52 - name: vmClass
53 value: best-effort-small
54 - name: storageClass
55 value: all-vsans #my zonal storageclass
Lets apply and see what happens. What I am expecting is the worker nodes should be placed according to the plan above, 1 worker pr vSphere cluster. The control plane node will be placed random.
1linuxvm01:~/three-zones$ k apply -f three-zone-cluster-1.yaml
2cluster.cluster.x-k8s.io/three-zone-cluster-1 created
And the results are in:
All three worker nodes were placed in their respective vSphere Zones (vSphere clusters) as configured in the yaml. The control plane node was just randomly placed in vSphere zone 3.
Thats it for this task. Now I want to deploy nearly the same, but with 3 control plane nodes. Where will they be placed?
Here is the cluster definition:
1apiVersion: cluster.x-k8s.io/v1beta1
2kind: Cluster
3metadata:
4 name: three-zone-cluster-1 #My own name on the cluster
5 namespace: ns-three-zone-1 #My vSphere Namespace
6spec:
7 clusterNetwork:
8 services:
9 cidrBlocks: ["20.30.0.0/16"] #Edited by me
10 pods:
11 cidrBlocks: ["20.40.0.0/16"] #Edited by me
12 serviceDomain: "cluster.local"
13 topology:
14 class: tanzukubernetescluster
15 version: v1.24.9+vmware.1-tkg.4 #My latest available TKR version
16 controlPlane:
17 replicas: 3 # should be spread evenly across zones
18 metadata:
19 annotations:
20 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
21 workers:
22 #muliple node pools are used
23 machineDeployments:
24 - class: node-pool
25 name: node-pool-1
26 replicas: 1 #only 1 worker here
27 metadata:
28 annotations:
29 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
30 #failure domain the machines will be created in
31 #maps to a vSphere Zone; name must match exactly
32 failureDomain: wdc-zone-1 #named after my vSphere zone
33 - class: node-pool
34 name: node-pool-2
35 replicas: 1 #only 1 worker here
36 metadata:
37 annotations:
38 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
39 #failure domain the machines will be created in
40 #maps to a vSphere Zone; name must match exactly
41 failureDomain: wdc-zone-2 #named after my vSphere zone
42 - class: node-pool
43 name: node-pool-3
44 replicas: 1 #only 1 worker here
45 metadata:
46 annotations:
47 run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
48 #failure domain the machines will be created in
49 #maps to a vSphere Zone; name must match exactly
50 failureDomain: wdc-zone-3 #named after my vSphere zone
51 variables:
52 - name: vmClass
53 value: best-effort-small
54 - name: storageClass
55 value: all-vsans #my zonal storageclass
And in vCenter where is the control plane nodes placed:
Application placement
After your workload cluster has been deployed as specified above you also want to utilize the different vSphere Zones for application placement. Start by switching into the context of your workload cluster:
1linuxvm01:~/three-zones$ kubectl vsphere login --server=10.101.90.2 --insecure-skip-tls-verify --vsphere-username=andreasm@cpod-nsxam-wdc.az-wdc.cloud-garage.net --tanzu-kubernetes-cluster-namespace ns-three-zone-1 --tanzu-ku
2bernetes-cluster-name three-zone-cluster-1
1linuxvm01:~/three-zones$ k config current-context
2three-zone-cluster-1
Now that I am in the correct context, lets check the nodes status, and specifically if there are any labels of interest or relevance to the vSphere Zones.
1linuxvm01:~/three-zones$ k get nodes --show-labels
2NAME STATUS ROLES AGE VERSION LABELS
3three-zone-cluster-1-h9nxh-9xsvp Ready control-plane 4m25s v1.24.9+vmware.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-2,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-h9nxh-9xsvp,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-2
4three-zone-cluster-1-h9nxh-bqqzd Ready control-plane 16m v1.24.9+vmware.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-3,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-h9nxh-bqqzd,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-3
5three-zone-cluster-1-h9nxh-kkvkz Ready control-plane 10m v1.24.9+vmware.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-1,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-h9nxh-kkvkz,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-1
6three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz Ready <none> 13m v1.24.9+vmware.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-1,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz,kubernetes.io/os=linux,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-1
7three-zone-cluster-1-node-pool-2-prg6m-84d45c4bd-vwhns Ready <none> 11m v1.24.9+vmware.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-2,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-node-pool-2-prg6m-84d45c4bd-vwhns,kubernetes.io/os=linux,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-2
8three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6 Ready <none> 11m v1.24.9+vmware.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-3,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6,kubernetes.io/os=linux,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-3
In both the worker nodes and control plane nodes we have the labels:
1linuxvm01:~/three-zones$ k get nodes --show-labels
2NAME LABELS
3three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz topology.kubernetes.io/zone=wdc-zone-1
4three-zone-cluster-1-node-pool-2-prg6m-84d45c4bd-vwhns topology.kubernetes.io/zone=wdc-zone-2
5three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6 topology.kubernetes.io/zone=wdc-zone-3
These labels can then be used when we deploy our applications. I will be using node affinity in my example below. For more information on pod to node placement see here.
Now I want to deploy an application called Yelb that consist of four pods.
I will define the yelp-db, yelp-appserver and yelb-cache pod to be placed in my vSphere Zone 3. The yelb-ui I will define to be placed in my vSphere Zone 1.
Below is my yelp application yaml manifest.
1apiVersion: v1
2kind: Service
3metadata:
4 name: redis-server
5 labels:
6 app: redis-server
7 tier: cache
8 namespace: yelb
9spec:
10 type: ClusterIP
11 ports:
12 - port: 6379
13 selector:
14 app: redis-server
15 tier: cache
16---
17apiVersion: v1
18kind: Service
19metadata:
20 name: yelb-db
21 labels:
22 app: yelb-db
23 tier: backenddb
24 namespace: yelb
25spec:
26 type: ClusterIP
27 ports:
28 - port: 5432
29 selector:
30 app: yelb-db
31 tier: backenddb
32---
33apiVersion: v1
34kind: Service
35metadata:
36 name: yelb-appserver
37 labels:
38 app: yelb-appserver
39 tier: middletier
40 namespace: yelb
41spec:
42 type: ClusterIP
43 ports:
44 - port: 4567
45 selector:
46 app: yelb-appserver
47 tier: middletier
48---
49apiVersion: v1
50kind: Service
51metadata:
52 name: yelb-ui
53 labels:
54 app: yelb-ui
55 tier: frontend
56 namespace: yelb
57spec:
58 type: LoadBalancer
59 ports:
60 - port: 80
61 protocol: TCP
62 targetPort: 80
63 selector:
64 app: yelb-ui
65 tier: frontend
66---
67apiVersion: apps/v1
68kind: Deployment
69metadata:
70 name: yelb-ui
71 namespace: yelb
72spec:
73 selector:
74 matchLabels:
75 app: yelb-ui
76 replicas: 1
77 template:
78 metadata:
79 labels:
80 app: yelb-ui
81 tier: frontend
82 spec:
83 affinity:
84 nodeAffinity:
85 requiredDuringSchedulingIgnoredDuringExecution:
86 nodeSelectorTerms:
87 - matchExpressions:
88 - key: topology.kubernetes.io/zone
89 operator: In
90 values:
91 - wdc-zone-1
92 containers:
93 - name: yelb-ui
94 image: registry.guzware.net/yelb/yelb-ui:0.3
95 imagePullPolicy: Always
96 ports:
97 - containerPort: 80
98---
99apiVersion: apps/v1
100kind: Deployment
101metadata:
102 name: redis-server
103 namespace: yelb
104spec:
105 selector:
106 matchLabels:
107 app: redis-server
108 replicas: 1
109 template:
110 metadata:
111 labels:
112 app: redis-server
113 tier: cache
114 spec:
115 affinity:
116 nodeAffinity:
117 requiredDuringSchedulingIgnoredDuringExecution:
118 nodeSelectorTerms:
119 - matchExpressions:
120 - key: topology.kubernetes.io/zone
121 operator: In
122 values:
123 - wdc-zone-3
124 containers:
125 - name: redis-server
126 image: registry.guzware.net/yelb/redis:4.0.2
127 ports:
128 - containerPort: 6379
129---
130apiVersion: apps/v1
131kind: Deployment
132metadata:
133 name: yelb-db
134 namespace: yelb
135spec:
136 selector:
137 matchLabels:
138 app: yelb-db
139 replicas: 1
140 template:
141 metadata:
142 labels:
143 app: yelb-db
144 tier: backenddb
145 spec:
146 affinity:
147 nodeAffinity:
148 requiredDuringSchedulingIgnoredDuringExecution:
149 nodeSelectorTerms:
150 - matchExpressions:
151 - key: topology.kubernetes.io/zone
152 operator: In
153 values:
154 - wdc-zone-3
155 containers:
156 - name: yelb-db
157 image: registry.guzware.net/yelb/yelb-db:0.3
158 ports:
159 - containerPort: 5432
160---
161apiVersion: apps/v1
162kind: Deployment
163metadata:
164 name: yelb-appserver
165 namespace: yelb
166spec:
167 selector:
168 matchLabels:
169 app: yelb-appserver
170 replicas: 1
171 template:
172 metadata:
173 labels:
174 app: yelb-appserver
175 tier: middletier
176 spec:
177 affinity:
178 nodeAffinity:
179 requiredDuringSchedulingIgnoredDuringExecution:
180 nodeSelectorTerms:
181 - matchExpressions:
182 - key: topology.kubernetes.io/zone
183 operator: In
184 values:
185 - wdc-zone-3
186 containers:
187 - name: yelb-appserver
188 image: registry.guzware.net/yelb/yelb-appserver:0.3
189 ports:
190 - containerPort: 4567
This is the section I define where the deployments should be placed:
1 affinity:
2 nodeAffinity:
3 requiredDuringSchedulingIgnoredDuringExecution:
4 nodeSelectorTerms:
5 - matchExpressions:
6 - key: topology.kubernetes.io/zone
7 operator: In
8 values:
9 - wdc-zone-X
And now I apply my application, and where will the different pods be placed:
1linuxvm01:~/three-zones$ k apply -f yelb-lb-zone-affinity.yaml
2service/redis-server created
3service/yelb-db created
4service/yelb-appserver created
5service/yelb-ui created
6deployment.apps/yelb-ui created
7deployment.apps/redis-server created
8deployment.apps/yelb-db created
9deployment.apps/yelb-appserver created
Check pod information with -o wide
1linuxvm01:~/three-zones$ k get pods -n yelb -o wide
2NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
3redis-server-6cc65b47bd-sndht 1/1 Running 0 70s 20.40.3.8 three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6 <none> <none>
4yelb-appserver-84d4784595-jw7m5 1/1 Running 0 70s 20.40.3.10 three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6 <none> <none>
5yelb-db-7f888657dd-nt427 1/1 Running 0 70s 20.40.3.9 three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6 <none> <none>
6yelb-ui-6597db5d9b-972h7 1/1 Running 0 70s 20.40.1.6 three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz <none> <none>
As I can see from this output all pods execpt my frontend-ui pod has been placed in vSphere Zone 3. Now, if you read the official documentation from Kubernets.io pod placemement can be done in different ways according to different needs. Worth reading.
This concludes this post.