vSphere with Tanzu and Multi Zones part2

Overview

This post will explore a bit further how we can utilize the three-zone setup configured here for TKG cluster and application placement. It will not be a very long post, but showing how it can be done and which values to use. In the "part 1" post I enabled a three-zone Supervisor deployment, meaning the three Supervisor Control Plane VMs will be distributed evenly across my three vSphere Clusters. To use a three-zone deployment we need to have three vSphere Zones defined. Each of these vSphere zones is described as a "Failure Domain" and becomes a value we can use when we decide the placement of both TKG clusters and applications inside our TKG clusters. So basically this post will describe how I can take advantage of this when I deploy my workload clusters and how can I decide where my applications will be placed.

Workload cluster placement

My Supervisor cluster is already deployed in my three vSphere Zones, just waiting for me to give it something to do. I have created a vSphere Namespace for my TKG cluster called ns-three-zone-1. I want to use a different workload network than my Supervisor workload network is placed on, that is a benefit when using NSX-T.

To give some context, this is how my environment is looking before deploying the TKG cluster:

Three-Zones

Now I just need to log into the supervisor, prepare my TKG yaml manifest and deploy my TKG cluster.

Log in to Supervisor

 1linuxvm01:~/$ kubectl vsphere login --server=10.101.90.2 --insecure-skip-tls-verify --vsphere-username=andreasm@cpod-nsxam-wdc.az-wdc.cloud-garage.net
 2
 3
 4KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
 5Password:
 6Logged in successfully.
 7
 8You have access to the following contexts:
 9   10.101.90.2
10
11If the context you wish to use is not in this list, you may need to try
12logging in again later, or contact your cluster administrator.
13
14To change context, use `kubectl config use-context <workload name>`
15linuxvm01:~/$

Now that I am logged in, I will check if my vSphere Zones are available by issuing the following command:

1linuxvm01:~/three-zones$ k get vspherezones.topology.tanzu.vmware.com
2NAME         AGE
3wdc-zone-1   19d
4wdc-zone-2   19d
5wdc-zone-3   19d
6linuxvm01:~/three-zones$

So it seems, now I need to use these names/labels in my TKG yaml manifest. In my first deployment I will use the example from the official VMware documentation here with some additions from my side like run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu I will be using the API v1beta1 with kubectl not Tanzu CLI.

Let us have a look at it and edit it accordingly:

 1apiVersion: cluster.x-k8s.io/v1beta1
 2kind: Cluster
 3metadata:
 4  name: three-zone-cluster-1 #My own name on the cluster
 5  namespace: ns-three-zone-1 #My vSphere Namespace
 6spec:
 7  clusterNetwork:
 8    services:
 9      cidrBlocks: ["20.30.0.0/16"] #Edited by me
10    pods:
11      cidrBlocks: ["20.40.0.0/16"] #Edited by me
12    serviceDomain: "cluster.local"
13  topology:
14    class: tanzukubernetescluster
15    version: v1.24.9+vmware.1-tkg.4 #My latest available TKR version
16    controlPlane:
17      replicas: 1 # only one controlplane (saving resources and time)
18      metadata:
19        annotations:
20          run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
21    workers:
22      #muliple node pools are used
23      machineDeployments:
24        - class: node-pool
25          name: node-pool-1
26          replicas: 1 #only 1 worker here
27          metadata:
28            annotations:
29              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
30          #failure domain the machines will be created in
31          #maps to a vSphere Zone; name must match exactly
32          failureDomain: wdc-zone-1 #named after my vSphere zone
33        - class: node-pool
34          name: node-pool-2
35          replicas: 1 #only 1 worker here
36          metadata:
37            annotations:
38              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
39          #failure domain the machines will be created in
40          #maps to a vSphere Zone; name must match exactly
41          failureDomain: wdc-zone-2 #named after my vSphere zone
42        - class: node-pool
43          name: node-pool-3
44          replicas: 1 #only 1 worker here
45          metadata:
46            annotations:
47              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
48          #failure domain the machines will be created in
49          #maps to a vSphere Zone; name must match exactly
50          failureDomain: wdc-zone-3 #named after my vSphere zone
51    variables:
52      - name: vmClass
53        value: best-effort-small
54      - name: storageClass
55        value: all-vsans #my zonal storageclass

Lets apply and see what happens. What I am expecting is the worker nodes should be placed according to the plan above, 1 worker pr vSphere cluster. The control plane node will be placed random.

1linuxvm01:~/three-zones$ k apply -f three-zone-cluster-1.yaml
2cluster.cluster.x-k8s.io/three-zone-cluster-1 created

And the results are in:

cluster-1

cluster-2

cluster-3

All three worker nodes were placed in their respective vSphere Zones (vSphere clusters) as configured in the yaml. The control plane node was just randomly placed in vSphere zone 3.

Thats it for this task. Now I want to deploy nearly the same, but with 3 control plane nodes. Where will they be placed?

Here is the cluster definition:

 1apiVersion: cluster.x-k8s.io/v1beta1
 2kind: Cluster
 3metadata:
 4  name: three-zone-cluster-1 #My own name on the cluster
 5  namespace: ns-three-zone-1 #My vSphere Namespace
 6spec:
 7  clusterNetwork:
 8    services:
 9      cidrBlocks: ["20.30.0.0/16"] #Edited by me
10    pods:
11      cidrBlocks: ["20.40.0.0/16"] #Edited by me
12    serviceDomain: "cluster.local"
13  topology:
14    class: tanzukubernetescluster
15    version: v1.24.9+vmware.1-tkg.4 #My latest available TKR version
16    controlPlane:
17      replicas: 3 # should be spread evenly across zones
18      metadata:
19        annotations:
20          run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
21    workers:
22      #muliple node pools are used
23      machineDeployments:
24        - class: node-pool
25          name: node-pool-1
26          replicas: 1 #only 1 worker here
27          metadata:
28            annotations:
29              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
30          #failure domain the machines will be created in
31          #maps to a vSphere Zone; name must match exactly
32          failureDomain: wdc-zone-1 #named after my vSphere zone
33        - class: node-pool
34          name: node-pool-2
35          replicas: 1 #only 1 worker here
36          metadata:
37            annotations:
38              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
39          #failure domain the machines will be created in
40          #maps to a vSphere Zone; name must match exactly
41          failureDomain: wdc-zone-2 #named after my vSphere zone
42        - class: node-pool
43          name: node-pool-3
44          replicas: 1 #only 1 worker here
45          metadata:
46            annotations:
47              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
48          #failure domain the machines will be created in
49          #maps to a vSphere Zone; name must match exactly
50          failureDomain: wdc-zone-3 #named after my vSphere zone
51    variables:
52      - name: vmClass
53        value: best-effort-small
54      - name: storageClass
55        value: all-vsans #my zonal storageclass

And in vCenter where is the control plane nodes placed:

vsphere zone-1

vsphere-zone-2

vsphere-zone-3

Application placement

After your workload cluster has been deployed as specified above you also want to utilize the different vSphere Zones for application placement. Start by switching into the context of your workload cluster:

1linuxvm01:~/three-zones$ kubectl vsphere login --server=10.101.90.2 --insecure-skip-tls-verify --vsphere-username=andreasm@cpod-nsxam-wdc.az-wdc.cloud-garage.net --tanzu-kubernetes-cluster-namespace ns-three-zone-1 --tanzu-ku
2bernetes-cluster-name three-zone-cluster-1
1linuxvm01:~/three-zones$ k config current-context
2three-zone-cluster-1

Now that I am in the correct context, lets check the nodes status, and specifically if there are any labels of interest or relevance to the vSphere Zones.

1linuxvm01:~/three-zones$ k get nodes --show-labels
2NAME                                                      STATUS   ROLES           AGE     VERSION            LABELS
3three-zone-cluster-1-h9nxh-9xsvp                          Ready    control-plane   4m25s   v1.24.9+vmware.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-2,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-h9nxh-9xsvp,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-2
4three-zone-cluster-1-h9nxh-bqqzd                          Ready    control-plane   16m     v1.24.9+vmware.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-3,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-h9nxh-bqqzd,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-3
5three-zone-cluster-1-h9nxh-kkvkz                          Ready    control-plane   10m     v1.24.9+vmware.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-1,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-h9nxh-kkvkz,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-1
6three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz   Ready    <none>          13m     v1.24.9+vmware.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-1,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz,kubernetes.io/os=linux,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-1
7three-zone-cluster-1-node-pool-2-prg6m-84d45c4bd-vwhns    Ready    <none>          11m     v1.24.9+vmware.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-2,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-node-pool-2-prg6m-84d45c4bd-vwhns,kubernetes.io/os=linux,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-2
8three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6    Ready    <none>          11m     v1.24.9+vmware.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=wdc-zone-3,kubernetes.io/arch=amd64,kubernetes.io/hostname=three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6,kubernetes.io/os=linux,run.tanzu.vmware.com/kubernetesDistributionVersion=v1.24.9---vmware.1-tkg.4,run.tanzu.vmware.com/tkr=v1.24.9---vmware.1-tkg.4,topology.kubernetes.io/zone=wdc-zone-3

In both the worker nodes and control plane nodes we have the labels:

1linuxvm01:~/three-zones$ k get nodes --show-labels
2NAME                                                      LABELS
3three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz   topology.kubernetes.io/zone=wdc-zone-1
4three-zone-cluster-1-node-pool-2-prg6m-84d45c4bd-vwhns    topology.kubernetes.io/zone=wdc-zone-2
5three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6    topology.kubernetes.io/zone=wdc-zone-3

These labels can then be used when we deploy our applications. I will be using node affinity in my example below. For more information on pod to node placement see here.

Now I want to deploy an application called Yelb that consist of four pods.

yelb-application

I will define the yelp-db, yelp-appserver and yelb-cache pod to be placed in my vSphere Zone 3. The yelb-ui I will define to be placed in my vSphere Zone 1.

Below is my yelp application yaml manifest.

  1apiVersion: v1
  2kind: Service
  3metadata:
  4  name: redis-server
  5  labels:
  6    app: redis-server
  7    tier: cache
  8  namespace: yelb
  9spec:
 10  type: ClusterIP
 11  ports:
 12  - port: 6379
 13  selector:
 14    app: redis-server
 15    tier: cache
 16---
 17apiVersion: v1
 18kind: Service
 19metadata:
 20  name: yelb-db
 21  labels:
 22    app: yelb-db
 23    tier: backenddb
 24  namespace: yelb
 25spec:
 26  type: ClusterIP
 27  ports:
 28  - port: 5432
 29  selector:
 30    app: yelb-db
 31    tier: backenddb
 32---
 33apiVersion: v1
 34kind: Service
 35metadata:
 36  name: yelb-appserver
 37  labels:
 38    app: yelb-appserver
 39    tier: middletier
 40  namespace: yelb
 41spec:
 42  type: ClusterIP
 43  ports:
 44  - port: 4567
 45  selector:
 46    app: yelb-appserver
 47    tier: middletier
 48---
 49apiVersion: v1
 50kind: Service
 51metadata:
 52  name: yelb-ui
 53  labels:
 54    app: yelb-ui
 55    tier: frontend
 56  namespace: yelb
 57spec:
 58  type: LoadBalancer
 59  ports:
 60  - port: 80
 61    protocol: TCP
 62    targetPort: 80
 63  selector:
 64    app: yelb-ui
 65    tier: frontend
 66---
 67apiVersion: apps/v1
 68kind: Deployment
 69metadata:
 70  name: yelb-ui
 71  namespace: yelb
 72spec:
 73  selector:
 74    matchLabels:
 75      app: yelb-ui
 76  replicas: 1
 77  template:
 78    metadata:
 79      labels:
 80        app: yelb-ui
 81        tier: frontend
 82    spec:
 83      affinity:
 84        nodeAffinity:
 85          requiredDuringSchedulingIgnoredDuringExecution:
 86            nodeSelectorTerms:
 87            - matchExpressions:
 88              - key: topology.kubernetes.io/zone
 89                operator: In
 90                values:
 91                - wdc-zone-1
 92      containers:
 93      - name: yelb-ui
 94        image: registry.guzware.net/yelb/yelb-ui:0.3
 95        imagePullPolicy: Always
 96        ports:
 97        - containerPort: 80
 98---
 99apiVersion: apps/v1
100kind: Deployment
101metadata:
102  name: redis-server
103  namespace: yelb
104spec:
105  selector:
106    matchLabels:
107      app: redis-server
108  replicas: 1
109  template:
110    metadata:
111      labels:
112        app: redis-server
113        tier: cache
114    spec:
115      affinity:
116        nodeAffinity:
117          requiredDuringSchedulingIgnoredDuringExecution:
118            nodeSelectorTerms:
119            - matchExpressions:
120              - key: topology.kubernetes.io/zone
121                operator: In
122                values:
123                - wdc-zone-3
124      containers:
125      - name: redis-server
126        image: registry.guzware.net/yelb/redis:4.0.2
127        ports:
128        - containerPort: 6379
129---
130apiVersion: apps/v1
131kind: Deployment
132metadata:
133  name: yelb-db
134  namespace: yelb
135spec:
136  selector:
137    matchLabels:
138      app: yelb-db
139  replicas: 1
140  template:
141    metadata:
142      labels:
143        app: yelb-db
144        tier: backenddb
145    spec:
146      affinity:
147        nodeAffinity:
148          requiredDuringSchedulingIgnoredDuringExecution:
149            nodeSelectorTerms:
150            - matchExpressions:
151              - key: topology.kubernetes.io/zone
152                operator: In
153                values:
154                - wdc-zone-3
155      containers:
156      - name: yelb-db
157        image: registry.guzware.net/yelb/yelb-db:0.3
158        ports:
159        - containerPort: 5432
160---
161apiVersion: apps/v1
162kind: Deployment
163metadata:
164  name: yelb-appserver
165  namespace: yelb
166spec:
167  selector:
168    matchLabels:
169      app: yelb-appserver
170  replicas: 1
171  template:
172    metadata:
173      labels:
174        app: yelb-appserver
175        tier: middletier
176    spec:
177      affinity:
178        nodeAffinity:
179          requiredDuringSchedulingIgnoredDuringExecution:
180            nodeSelectorTerms:
181            - matchExpressions:
182              - key: topology.kubernetes.io/zone
183                operator: In
184                values:
185                - wdc-zone-3
186      containers:
187      - name: yelb-appserver
188        image: registry.guzware.net/yelb/yelb-appserver:0.3
189        ports:
190        - containerPort: 4567

This is the section I define where the deployments should be placed:

1      affinity:
2        nodeAffinity:
3          requiredDuringSchedulingIgnoredDuringExecution:
4            nodeSelectorTerms:
5            - matchExpressions:
6              - key: topology.kubernetes.io/zone
7                operator: In
8                values:
9                - wdc-zone-X

And now I apply my application, and where will the different pods be placed:

1linuxvm01:~/three-zones$ k apply -f yelb-lb-zone-affinity.yaml
2service/redis-server created
3service/yelb-db created
4service/yelb-appserver created
5service/yelb-ui created
6deployment.apps/yelb-ui created
7deployment.apps/redis-server created
8deployment.apps/yelb-db created
9deployment.apps/yelb-appserver created

Check pod information with -o wide

1linuxvm01:~/three-zones$ k get pods -n yelb -o wide
2NAME                              READY   STATUS    RESTARTS   AGE   IP           NODE                                                      NOMINATED NODE   READINESS GATES
3redis-server-6cc65b47bd-sndht     1/1     Running   0          70s   20.40.3.8    three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6    <none>           <none>
4yelb-appserver-84d4784595-jw7m5   1/1     Running   0          70s   20.40.3.10   three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6    <none>           <none>
5yelb-db-7f888657dd-nt427          1/1     Running   0          70s   20.40.3.9    three-zone-cluster-1-node-pool-3-b6hkw-df698b86d-8hdd6    <none>           <none>
6yelb-ui-6597db5d9b-972h7          1/1     Running   0          70s   20.40.1.6    three-zone-cluster-1-node-pool-1-7xsnp-75994d44d8-zxzsz   <none>           <none>

As I can see from this output all pods execpt my frontend-ui pod has been placed in vSphere Zone 3. Now, if you read the official documentation from Kubernets.io pod placemement can be done in different ways according to different needs. Worth reading.

This concludes this post.