vSphere 8 with Tanzu using VDS and Avi Loadbalancer

Overview

Deploy Tanzu in vSphere 8 with VDS and Avi Loadbalancer:

This post will go through how to install Tanzu in vSphere 8 using vSphere VDS networking and Avi as loadbalancer. The goal is to deploy Tanzu by using vSphere Distributed Switch (no NSX this time) and utilize Avi as loadbalancer for Supervisor and workload cluster L4 endpoint (kubernetes API). When that is done I will go through how we also can extend this into L7 (Ingress) by using AKO in our workload clusters.

The below diagram is what we should end up with after the basic deployment of Tanzu and Avi:

Assumptions

This post assumes we already have a vSphere environment up and running with vCenter, HA and DRS. Required network to support the basic vSphere stuff like vMotion and shared storage. And the hosts networking has been configured with a Distributed Switch with the corresponding vds portgroups for Management, Frontend network (VIP placement for kubernetes API endpoint) and workload network with corresponding VLANs. In vCenter a content library needs to be created, this is just a local library you give a meaningful name no subscriptions etc.

At least one Avi controller is deployed, no cloud added, just deployed and the initial configs done.

Preparations on the Avi side of things

This part of the guide takes place on the newly configured Avi controller(s) which currently only has the initial configuration done

Avi cloud configurations

To prepare Avi for this deployment we need to configure the vCenter cloud. This is done here:

There is a Default-Cloud object there we need to convert to a vCenter cloud. This is done by clicking on this button on the far right side:

This will bring up the following options:

Select VMware vCenter/vSphere NSX Then start populate the relevant vCenter information for your vCenter:

When credentials is added slect content library and choose your content library from the list, then click connect and then Save & Relaunch

When the dialog relaunches select the management network and ip address management. I have opted for DHCP (I have DHCP in my mgmt network, if not we can leverage Avi as IPAM provider for the mgmt network also.) This is used for the SE's mgmt interface.

Click save for now. Head over to the Template section to add IPAM and DNS.

When using vCenter clouds the different portgroups is automatically added under networks. We need to configure some of them. But for now just create the IPAM and DNS profiles and we configure the networks later accordingly.

The DNS Profile (optional, only if you want to use Avi DNS service):

Click save when done

The IPAM profile:

Select the Default-Cloud, then select from the list "Usable Networks" the Frontend network vds portgroup corresponding to the frontend network we want to use for our endpoint vips. Click save. You should have your profiles configured now:

Head back to your cloud again and add your newly created IPAM and DNS profiles.

Add the profiles:

Before you click finish, make sure you have selected "Prefer Static Routes vs Directly Connected Network" like this:

Then click finish...

Avi network configs

Now its time to configure the networks for the SEs (VIP, dataplane). I will go ahead and configure both the Frontend VIP for kubernetes API endpoint, but also the workload network I will use when I add L7 functionality later. Head over to Cloud Resources:

Scroll until you find your "Frontend-network" Click edit all the way to the right:

In here we need to define the subnet (if not already auto discovered) and add and IP range for the SE's and VIPS. We can decide to create just one range for both, or a range only for SEs, and one for VIPs only. To create a range for both SE and VIP do like this:

If you want a specific range for SE and a specific for VIP do like this:

Common for both is to deselect the DHCP Enabled option. What we have done now is to tell Avi that Avi is responsible for IP allocation to our SE's when they are deployed and configured to give them each their IP in the Frontend network, and also "carve" out an IP for the VIP when a Virtual Service is created and IP allocation for that service is selected to auto-allocate. The same would go if you decided to not use DHCP for mgmt IP, you would need to defined the network and select only "Use for Service Engine" (no VIPs in the management network)

Avi service engine group

Now its time to prepare the Default Service Engine Group. Head over to Cloud Resources - Service Engine Group

Click the pencil on the far right side of the Default-Group and make the following changes:

In the Advanced tab:

Here we select our vSphere cluster for the SE placement, vSphere shared storage, and the Prefix and vCenter folder placement (if you want). Now that is done.

Avi VRF context

Now we need to create a static route for the SE dataplane to know which gateway will take them to the "backend-pool" (the services they are acting as loadbalancer for). This is usually the gateway for the networks in their respective subnet as the dataplane is residing in. Here I prepare the route for the Frontend network, and also the Workload network (so it is already done when moving to the step of enabling L7).

Avi controller SSL certificate for Tanzu "integration"

The last step is to create a new certificate, or use your own signed certificate, for the Tanzu deployment to use. Head over to Templates - SSL/TLS Certificates.

From here we click "Create" in the top right corner:

I will go ahead and create a new self-signed certificate:

It is important that you use the IP or FQDN of the controller under "Common Name" and under "Subject Alternate Name (SAN)"

Now head over to Administration - Access Settings:

Click edit on the pencil in the top right corner, remove the existing certificates under SSL/TLS Certificate:

And replace with the one you created:

Now the Avi config is done for this round. Next step is to enable Workload Management in vSphere...

Enable Workload Management in vSphere

This section will cover all the steps to enable Tanzu from vCenter, describing the selections made and the network configs.

Enable workload management

Head over to your vCenter server and click here: (from the the "hamburger menu" top left corner)

Click the Get Started button:

Step 1: Select vSphere Distributed Switch Step 2: Select Cluster Deployment, give the supervisor cluster a name and give it a zone name:

Step 3: Select your Storage Policy (If VSAN and you dont have created a specific VSAN policy for this use Default Storage Policy):

Step 4: Type in the relevant info for your Avi Controller and copy paste the certificate from your Avi controller:

The certificate is easily copied from the Avi controller by going to Templates - SSL/TLS Certificates and click the "down arrow":

Then copy the certificate:

Paste the content in the Server Certificate field above (step 4) Step 5: Management Network Here we fill in the required information for the Supervisor nodes. Management IP for the nodes themselves (needs connectivity to both vCenter and ESXi hosts, could be in the same mgmt network as vCenter and ESXi). Select the corresponding vds portgroup, select either static or DHCP if you want to use DHCP. Step 6: Workload Network

Select the correct vds portgroup for the workload network. The supervisor and the workload nodes will be placed here. Can be static or DHCP. Leave the default "Internal Network for Kubernetes Services", that is for the internal services (clusterIP etc inside the K8s clusters, they will never be exposed outside). Fill in the necessary config if you go with static. Step 7: Review and Confirm and optionally give the Supervisor endpoint a DNS name which you later can register in your DNS service when we have the L4 IP for the kubernetes API endpoint. Click finish:

The whole summary:

Now sit back and wait for the creation of the supervisor cluster, it can take a couple of minutes.

After a while you can take a look in your Avi controller under Applications and see if something is being created there;

You can monitor the process from the Workload management status view by clicking on the "Configuring (View) )". You can continue work with your vCenter server and go back to this progress bar whenever you want by clicking the hamburger menu Workload management.

In your vCenter inventory you should also see the Supervisor VMs and Avi SE's like this:

When its done deploying you will see the green status here:

Notice the Control Plane Node Address, this is our endpoint IP being served by Avi loadbalancer and the one we use to interact with the Supervisor cluster to create workloads etc..

Enter the IP in your browser with https://10.102.7.11 and you should see this page:

Download the cli tools for your operating system, deploy them so they are in your path. Will use both kubectl and kubectl-vsphere later on Next steps include creating namespace, deploy workload clusters. Lets dive into it.

vSphere Namespace

vSphere with Tanzu workloads, including vSphere Pods, VMs, and Tanzu Kubernetes clusters, are deployed to a vSphere Namespace. You define a vSphere Namespace on a Supervisor and configure it with resource quota and user permissions. Depending on the DevOps needs and workloads they plan to run, you might also assign storage policies, VM classes, and content libraries for fetching the latest Tanzu Kubernetes releases and VM images. source

Create a vSphere namespace

Now that the Supervisor cluster is ready and running head back to your vCenter and create a vSphere namespace.

Click create namespace (above) the select the supervisor to create the namespace on, and give your namespace a name then select the "workload network" you have defined for your workload placement.

Now the namespace is being created.

Add additional workload networks

Sidenote there is also possible to add more "workload networks" after the Supervisor has been configured under Supervisor config if you want to add more "workload networks" for separation etc. To do that head over to Workload Management in vCenter:

Then select the supervisor tab:

Click on your supervisor cluster here:

Then click the configure tab and go to network and add your additional workload network:

After your namespace has been created we need to configure it with access permissions, datastores, content library and vmclasses:

Create workload cluster

Afte the vSphere Namespace has been configured its time to deploy a workload cluster/TKC cluster (Tanzu Kubernetes Cluster cluster 😄 ). From your workstation/jumphost where you downloaded the cli tools login in to the supervisor with access rights to the Supervisor API. (administrator@vsphere.local will have access).

Custom role in vCenter

I created a specific role in my vCenter with these privileges:

Added my "supervisor-manager" user in this role as global and in the top tree of my vCenter with inheritance. Also added it as "editor" in my wdc-2-ns-1 vSphere Namespace.

1kubectl vsphere login --server=10.102.7.11 --insecure-skip-tls-verify --vsphere-username=supervisor-manager@cpod-nsxam-wdc.az-wdc.cloud-garage.net

When your are logged in it will give you this output and also put the kubernetes config in your ~/.kube/config file.

 1andreasm@linuxvm01:~$ kubectl vsphere login --server=10.102.7.11 --insecure-skip-tls-verify --vsphere-username=supervisor-manager@cpod-nsxam-wdc.az-wdc.cloud-garage.net
 2
 3
 4KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
 5Password: 
 6Logged in successfully.
 7
 8You have access to the following contexts:
 9   10.102.7.11
10   wdc-2-ns-1
11
12If the context you wish to use is not in this list, you may need to try
13logging in again later, or contact your cluster administrator.
14
15To change context, use `kubectl config use-context <workload name>`

When you are logged in prepare your yaml for your first workload cluster and apply it with kubectl apply -f nameof.yaml

Example:

 1apiVersion: cluster.x-k8s.io/v1beta1
 2kind: Cluster
 3metadata:
 4  name: wdc-2-tkc-cluster-1  # give your tkc cluster a name
 5  namespace: wdc-2-ns-1 # remember to put it in your defined vSphere Namespace
 6spec:
 7  clusterNetwork:
 8    services:
 9      cidrBlocks: ["20.10.0.0/16"]
10    pods:
11      cidrBlocks: ["20.20.0.0/16"]
12    serviceDomain: "cluster.local"
13  topology:
14    class: tanzukubernetescluster
15    version: v1.23.8---vmware.2-tkg.2-zshippable
16    controlPlane:
17      replicas: 1
18      metadata:
19        annotations:
20          run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
21    workers:
22      machineDeployments:
23        - class: node-pool
24          name: node-pool-01
25          replicas: 3
26          metadata:
27            annotations:
28              run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu
29    variables:
30      - name: vmClass
31        value: best-effort-medium
32      - name: storageClass
33        value: vsan-default-storage-policy

As soon as I apply the above yaml it will deploy the corresponding tkc cluster in your vsphere environment:

Sit back and enjoy while your tkc cluster is being created for you. We can check the status in the vCenter gui:

or via kubectl:

1andreasm@linuxvm01:~/tkc-wdc-01-vds$ k get cluster -n wdc-2-ns-1
2NAME                  PHASE         AGE   VERSION
3wdc-2-tkc-cluster-1   Provisioned   12m   v1.23.8+vmware.2

It is ready, now we need to log into it:

 1andreasm@linuxvm01:~/tkc-wdc-01-vds$ kubectl vsphere login --server=10.102.7.11 --insecure-skip-tls-verify --vsphere-username=supervisor-manager@cpod-nsxam-wdc.az-wdc.cloud-garage.net --tanzu-kubernetes-cluster-name=wdc-2-tkc-cluster-1 --tanzu-kubernetes-cluster-namespace=wdc-2-ns-1
 2
 3
 4KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
 5Password: 
 6Logged in successfully.
 7
 8You have access to the following contexts:
 9   10.102.7.11
10   wdc-2-ns-1
11   wdc-2-tkc-cluster-1
12
13If the context you wish to use is not in this list, you may need to try
14logging in again later, or contact your cluster administrator.
15
16To change context, use `kubectl config use-context <workload name>`

Check if you are able to list ns and pods:

 1andreasm@linuxvm01:~/tkc-wdc-01-vds$ k get pods -A
 2NAMESPACE                      NAME                                                                      READY   STATUS    RESTARTS        AGE
 3kube-system                    antrea-agent-77drs                                                        2/2     Running   0               8m35s
 4kube-system                    antrea-agent-j482r                                                        2/2     Running   0               8m34s
 5kube-system                    antrea-agent-thh5b                                                        2/2     Running   0               8m35s
 6kube-system                    antrea-agent-tz4fb                                                        2/2     Running   0               8m35s
 7kube-system                    antrea-controller-575845467f-pqgll                                        1/1     Running   0               8m35s
 8kube-system                    coredns-7d8f74b498-ft7rf                                                  1/1     Running   0               10m
 9kube-system                    coredns-7d8f74b498-pqgp7                                                  1/1     Running   0               7m35s
10kube-system                    docker-registry-wdc-2-tkc-cluster-1-n89g2-zc27k                           1/1     Running   0               11m
11kube-system                    docker-registry-wdc-2-tkc-cluster-1-node-pool-01-4dql7-7b8b84fb4b-6cvz9   1/1     Running   0               8m46s
12kube-system                    docker-registry-wdc-2-tkc-cluster-1-node-pool-01-4dql7-7b8b84fb4b-rgn29   1/1     Running   0               8m34s
13kube-system                    docker-registry-wdc-2-tkc-cluster-1-node-pool-01-4dql7-7b8b84fb4b-rsmfw   1/1     Running   0               9m
14kube-system                    etcd-wdc-2-tkc-cluster-1-n89g2-zc27k                                      1/1     Running   0               11m
15kube-system                    kube-apiserver-wdc-2-tkc-cluster-1-n89g2-zc27k                            1/1     Running   0               11m
16kube-system                    kube-controller-manager-wdc-2-tkc-cluster-1-n89g2-zc27k                   1/1     Running   0               11m
17kube-system                    kube-proxy-67xjk                                                          1/1     Running   0               8m46s
18kube-system                    kube-proxy-6fttt                                                          1/1     Running   0               8m35s
19kube-system                    kube-proxy-m4wt8                                                          1/1     Running   0               11m
20kube-system                    kube-proxy-rbsjw                                                          1/1     Running   0               9m1s
21kube-system                    kube-scheduler-wdc-2-tkc-cluster-1-n89g2-zc27k                            1/1     Running   0               11m
22kube-system                    metrics-server-6f7c489795-scmm6                                           1/1     Running   0               8m36s
23secretgen-controller           secretgen-controller-6966677567-4hngd                                     1/1     Running   0               8m26s
24tkg-system                     kapp-controller-55f9977c86-bqppj                                          2/2     Running   0               9m22s
25tkg-system                     tanzu-capabilities-controller-manager-cb4bc7978-qh9s8                     1/1     Running   3 (87s ago)     7m54s
26vmware-system-auth             guest-cluster-auth-svc-r4rk9                                              1/1     Running   0               7m48s
27vmware-system-cloud-provider   guest-cluster-cloud-provider-859b8dc577-8jlth                             1/1     Running   0               8m48s
28vmware-system-csi              vsphere-csi-controller-6db86b997-l5glc                                    6/6     Running   0               8m46s
29vmware-system-csi              vsphere-csi-node-7bdpl                                                    3/3     Running   2 (7m34s ago)   8m34s
30vmware-system-csi              vsphere-csi-node-q7zqr                                                    3/3     Running   3 (7m32s ago)   8m46s
31vmware-system-csi              vsphere-csi-node-r8v2c                                                    3/3     Running   3 (7m34s ago)   8m44s
32vmware-system-csi              vsphere-csi-node-zl4m2                                                    3/3     Running   3 (7m40s ago)   8m46s
 1andreasm@linuxvm01:~/tkc-wdc-01-vds$ k get ns
 2NAME                           STATUS   AGE
 3default                        Active   12m
 4kube-node-lease                Active   12m
 5kube-public                    Active   12m
 6kube-system                    Active   12m
 7secretgen-controller           Active   9m3s
 8tkg-system                     Active   9m55s
 9vmware-system-auth             Active   12m
10vmware-system-cloud-provider   Active   10m
11vmware-system-csi              Active   10m
12vmware-system-tkg              Active   12m

By default we are not allowed to run anything on our newly created tkc cluster. We need to define some ClusterRoles. I will just apply a global clusterolres on my tkc cluster so I can do what I want with it like this: Apply the psp policy yaml:

 1apiVersion: rbac.authorization.k8s.io/v1
 2kind: ClusterRole
 3metadata:
 4  name: psp:privileged
 5rules:
 6- apiGroups: ['policy']
 7  resources: ['podsecuritypolicies']
 8  verbs:     ['use']
 9  resourceNames:
10  - vmware-system-privileged
11---
12apiVersion: rbac.authorization.k8s.io/v1
13kind: ClusterRoleBinding
14metadata:
15  name: all:psp:privileged
16roleRef:
17  kind: ClusterRole
18  name: psp:privileged
19  apiGroup: rbac.authorization.k8s.io
20subjects:
21- kind: Group
22  name: system:serviceaccounts
23  apiGroup: rbac.authorization.k8s.io
1kubectl apply -f roles.yaml
2clusterrole.rbac.authorization.k8s.io/psp:privileged created
3clusterrolebinding.rbac.authorization.k8s.io/all:psp:privileged created

Now that I am allowed to deploy stuff I am ready to consume the newly cluster. But this blog was how to deploy Tanzu with VDS and Avi Loadbalancer. So far I have only covered the L4 part where Avi is providing me the K8s API endpoints, I will now jump over to the section where I configure both Avi and my tkc cluster to use Ingress (L7) also so I can publish/expose my applications with ingress. That means installing an additional component called AKO in my tkc cluster and configure Avi accordingly.

Configure Avi as Ingress controller (L7)

For Avi Ingress we need to deploy a component in our TKC cluster called AKO. AKO stands for Avi Kubernetes Operator and introduces the ability to translate our k8s api to the Avi controller so we can make our Avi automatically create vs services for us as soon as we request them from our TKC cluster. To deploy AKO we use Helm. In short we need to add the AKO helm repository, get the ako values, edit them to fit our environment, then install it by using Helm. So let us go through this step-by-step (I have also covered it a while back in an Upstream k8s cluster) but let us do it again here.

Create the namespace for the ako pod:

1k create ns avi-system
2namespace/avi-system created

Then add the repo to Helm:

1helm repo add ako https://projects.registry.vmware.com/chartrepo/ako

Check the repo:

1andreasm@linuxvm01:~$ helm search repo
2NAME            	CHART VERSION	APP VERSION	DESCRIPTION                             
3ako/ako         	1.8.2        	1.8.2      	A helm chart for Avi Kubernetes Operator
4ako/ako-operator	1.3.1        	1.3.1      	A Helm chart for Kubernetes AKO Operator
5ako/amko        	1.8.2        	1.8.2      	A helm chart for Avi Kubernetes Operator

Get the values.yaml:

1 helm show values ako/ako --version 1.8.2 > values.yaml

Now its time to edit the value file. I will go through the values files, update it accordingly and adjust some configurations in Avi controller.

The values.yaml for ako chart:

 1# this file has been edited by me to easier reflect the changes I have done. So all default comments have been removed, and contains only my comments.
 2replicaCount: 1
 3
 4image:
 5  repository: projects.registry.vmware.com/ako/ako
 6  pullPolicy: IfNotPresent
 7
 8
 9AKOSettings:
10  primaryInstance: true 
11  enableEvents: 'true'  
12  logLevel: WARN   
13  fullSyncFrequency: '1800' 
14  apiServerPort: 8080 
15  deleteConfig: 'false' 
16  disableStaticRouteSync: 'false' 
17  clusterName: wdc-tkc-cluster-1   # Here we need to define a name for our specific TKC cluster. This must not be the exact names as the cluster itself, but why not, it MUST be unique across all your TKC clusters if you have multiple AKO enabled TKC/K8s clusters on same Avi controller
18  cniPlugin: 'antrea' #This needs to be set to Antrea, the reason is that you would like to configure NodePortLocal.
19  enableEVH: false 
20  layer7Only: true # This is very important to set to true as we already have an other AKO instance managing L4 for our k8s api endpoints. We will only configure this instance to use L7. 
21  
22  namespaceSelector:
23    labelKey: ''
24    labelValue: ''
25  servicesAPI: false 
26  vipPerNamespace: 'false' 
27
28NetworkSettings:
29  nodeNetworkList:
30  # nodeNetworkList:
31    - networkName: "vds-tkc-workload-vlan-1026" # this is the VDS portgroup you have for your specific TKC cluster workers running in. In my case the defined portgroup name above. You can see this in vCenter
32      cidrs:
33      - 10.102.6.0/24 # this is the CIDR for your workers
34  enableRHI: false 
35  nsxtT1LR: '' 
36  bgpPeerLabels: [] 
37  # bgpPeerLabels:
38  #   - peer1
39  #   - peer2
40  vipNetworkList:  
41    - networkName: "vds-tkc-frontend-vlan-1027" # This can be the same VIP network you have configured in previously for api endpint or it can be a completely new one. I am going the easy route using the same. It can be shared across multiple cluster (is using NodePortLocal), or can be specific for each tkc cluster. 
42      cidr: 10.102.7.0/24
43
44L7Settings:
45  defaultIngController: 'true' # Specify if this is the only Ingress controller you have or default if using several others.
46  noPGForSNI: false 
47  serviceType: NodePortLocal # Here we select nodeportlocal - verify that Antrea is configured to use NodePortLocal
48  shardVSSize: SMALL   # I am setting this to small so I can run more services using same IP.
49  passthroughShardSize: SMALL   
50  enableMCI: 'false' 
51
52L4Settings:
53  defaultDomain: '' 
54  autoFQDN: default   
55
56
57ControllerSettings:
58  serviceEngineGroupName: Default-Group   # If you dont decide to use the same VIP as k8s api endpoint you could decide to create an additional ServiceEngineGroup for your L7 services (IP separation etc). 
59  controllerVersion: '22.1.1' # AKO version 1.8.2 supports Avi 22.1.3
60  cloudName: Default-Cloud   # The configured cloud name on the Avi controller.
61  controllerHost: '172.21.101.50' # IP address or Hostname of Avi Controller
62  tenantName: admin   
63
64nodePortSelector: 
65  key: ''
66  value: ''
67
68resources:
69  limits:
70    cpu: 350m
71    memory: 400Mi
72  requests:
73    cpu: 200m
74    memory: 300Mi
75
76podSecurityContext: {}
77
78rbac:
79  pspEnable: false
80
81
82avicredentials:
83  username: 'admin' # username for the Avi controller
84  password: 'password' # password for the Avi controller
85  authtoken:
86  certificateAuthorityData:
87
88
89persistentVolumeClaim: ''
90mountPath: /log
91logFile: avi.log

A word around the VIP network used for the L7/Ingress. As we deploy AKO as standalone we are not restricted to use only the components defined to support the install of Tanzu with vSphere, like service engine groups, vip networks etc. We could decide to create a separate VIP network by using a dedicated SE group for these networks. We could also decide to have the SE's using a separate dataplane network than the VIP itself. If going this path there is some config steps that needs to be taken on the network side. Routing to the VIP addresses, either Avi can be configured by using BGP, or we create static routes in the physical routers. But as the VIPs are coming and going (applications are published, deleted, etc) these IPs change. So BGP would be the best option, or use an already defined VLAN as I am doing in this example. In my other post on using NSX and Avi with Tanzu I will show how to use NSX for BGP. Maybe I will update this post also by adding a section where I use BGP from Avi to my upstream router. But for now I will stick with using my VLAN I have called frontend which already have a gateway and a route defined. So all my VIPs will be reachable through this network.

Antrea NodePortLocal

And another word around NodePortLocal. To be able to utilize NodePortLocal your Antrea config in the TKC cluster must be verified whether it is configured with NPL or not. So let us do instead of just assume something.

 1andreasm@linuxvm01:~/ako/ako_vds$ k get configmaps -n kube-system antrea-config-f5d8g47b88 -oyaml
 2apiVersion: v1
 3data:
 4  antrea-agent.conf: |
 5    featureGates:
 6      AntreaProxy: true
 7      EndpointSlice: true
 8      Traceflow: true
 9      NodePortLocal: false
10      AntreaPolicy: true
11      FlowExporter: false
12      NetworkPolicyStats: false
13      Egress: false
14      AntreaIPAM: false
15      Multicast: false
16      ServiceExternalIP: false
17    trafficEncapMode: encap
18    noSNAT: false
19    tunnelType: geneve
20    trafficEncryptionMode: none
21    wireGuard:
22      port: 51820
23    egress: {}
24    serviceCIDR: 20.10.0.0/16

Well that was not good. So we need to enable it. Luckily, with Tanzu with vSphere its quite simple actually. Switch context to your vSphere Namespace, edit an antreaconfig, apply it.

1andreasm@linuxvm01:~/antrea$ k config use-context wdc-2-ns-1 
2Switched to context "wdc-2-ns-1".
3andreasm@linuxvm01:~/antrea$ k get cluster
4NAME                  PHASE         AGE     VERSION
5wdc-2-tkc-cluster-1   Provisioned   3h26m   v1.23.8+vmware.2
6andreasm@linuxvm01:~/antrea$ k apply -f antreaconfig-wdc-2-nsx-1.yaml 
7Warning: resource antreaconfigs/wdc-2-tkc-cluster-1-antrea-package is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
8antreaconfig.cni.tanzu.vmware.com/wdc-2-tkc-cluster-1-antrea-package configured

The antreaconfig I used:

 1apiVersion: cni.tanzu.vmware.com/v1alpha1
 2kind: AntreaConfig
 3metadata:
 4  name: wdc-2-tkc-cluster-1-antrea-package  # notice the naming-convention tkc cluster name-antrea-package
 5  namespace: wdc-2-ns-1 # your vSphere Namespace the TKC cluster is in.
 6spec:
 7  antrea:
 8    config:
 9      featureGates:
10        AntreaProxy: true
11        EndpointSlice: false
12        AntreaPolicy: true
13        FlowExporter: false
14        Egress: true
15        NodePortLocal: true # Set this to true
16        AntreaTraceflow: true
17        NetworkPolicyStats: true

Lets have a look at my Antrea config in my TKC cluster now:

 1andreasm@linuxvm01:~/antrea$ k config use-context 
 210.102.7.11          wdc-2-ns-1           wdc-2-tkc-cluster-1  
 3andreasm@linuxvm01:~/antrea$ k config use-context wdc-2-tkc-cluster-1 
 4Switched to context "wdc-2-tkc-cluster-1".
 5andreasm@linuxvm01:~/antrea$ k get configmaps -n kube-system antrea-config-f5d8g47b88 -oyaml
 6apiVersion: v1
 7data:
 8  antrea-agent.conf: |
 9    featureGates:
10      AntreaProxy: true
11      EndpointSlice: false
12      Traceflow: true
13      NodePortLocal: true # Yes, there it is
14      AntreaPolicy: true
15      FlowExporter: false
16      NetworkPolicyStats: true
17      Egress: true
18      AntreaIPAM: false
19      Multicast: false
20      ServiceExternalIP: false
21    trafficEncapMode: encap
22    noSNAT: false
23    tunnelType: geneve
24    trafficEncryptionMode: none
25    wireGuard:
26      port: 51820
27    egress:
28      exceptCIDRs: []
29    serviceCIDR: 20.10.0.0/16

But... Even though our cluster config has been updated we need to delete the Antrea pods so they can restart and read their new configmap again.

 1andreasm@linuxvm01:~/ako$ k delete pod -n kube-system antrea-controller-575845467f-pqgll 
 2pod "antrea-controller-575845467f-pqgll" deleted
 3andreasm@linuxvm01:~/ako$ k delete pod -n kube-system antrea-agent-
 4antrea-agent-77drs  antrea-agent-j482r  antrea-agent-thh5b  antrea-agent-tz4fb  
 5andreasm@linuxvm01:~/ako$ k delete pod -n kube-system antrea-agent-77drs 
 6pod "antrea-agent-77drs" deleted
 7andreasm@linuxvm01:~/ako$ k delete pod -n kube-system antrea-agent-j482r 
 8pod "antrea-agent-j482r" deleted
 9andreasm@linuxvm01:~/ako$ k delete pod -n kube-system antrea-agent-thh5b 
10pod "antrea-agent-thh5b" deleted
11andreasm@linuxvm01:~/ako$ k delete pod -n kube-system antrea-agent-tz4fb 
12pod "antrea-agent-tz4fb" deleted

Configure Avi as Ingress controller (L7) - continue

Now that we have assured NodePortLocal is configured, its time to deploy AKO. I have also verified that I have the VIP network configured in Avi as I am using the existing network that is already defined. So install AKO then 😄

1helm install  ako/ako  --generate-name --version 1.8.2 -f ako.vds.wdc-2.values.yaml --namespace=avi-system
2NAME: ako-1675511021
3LAST DEPLOYED: Sat Feb  4 11:43:42 2023
4NAMESPACE: avi-system
5STATUS: deployed
6REVISION: 1
7TEST SUITE: None

Verify that the AKO pod is running:

1andreasm@linuxvm01:~/ako/ako_vds$ k get pods -n avi-system 
2NAME    READY   STATUS    RESTARTS   AGE
3ako-0   1/1     Running   0          57s

Check logs for some immediate messages that needs investigating before trying to deploy a test application.

 1andreasm@linuxvm01:~/ako/ako_vds$ k logs -n avi-system ako-0 
 22023-02-04T11:43:51.240Z	INFO	api/api.go:52	Setting route for GET /api/status
 32023-02-04T11:43:51.241Z	INFO	ako-main/main.go:71	AKO is running with version: v1.8.2
 42023-02-04T11:43:51.241Z	INFO	api/api.go:110	Starting API server at :8080
 52023-02-04T11:43:51.242Z	INFO	ako-main/main.go:81	We are running inside kubernetes cluster. Won't use kubeconfig files.
 62023-02-04T11:43:51.265Z	INFO	lib/control_config.go:198	ako.vmware.com/v1alpha1/AviInfraSetting enabled on cluster
 72023-02-04T11:43:51.270Z	INFO	lib/control_config.go:207	ako.vmware.com/v1alpha1/HostRule enabled on cluster
 82023-02-04T11:43:51.273Z	INFO	lib/control_config.go:216	ako.vmware.com/v1alpha1/HTTPRule enabled on cluster
 92023-02-04T11:43:51.290Z	INFO	ako-main/main.go:150	Kubernetes cluster apiserver version 1.23
102023-02-04T11:43:51.296Z	INFO	utils/utils.go:168	Initializing configmap informer in avi-system
112023-02-04T11:43:51.296Z	INFO	lib/dynamic_client.go:118	Skipped initializing dynamic informers antrea 
122023-02-04T11:43:51.445Z	INFO	k8s/ako_init.go:455	Successfully connected to AVI controller using existing AKO secret
132023-02-04T11:43:51.446Z	INFO	ako-main/main.go:261	Valid Avi Secret found, continuing .. 
142023-02-04T11:43:51.866Z	INFO	cache/avi_ctrl_clients.go:71	Setting the client version to 22.1.1
152023-02-04T11:43:51.866Z	INFO	ako-main/main.go:279	SEgroup name found, continuing ..
162023-02-04T11:43:53.015Z	INFO	cache/controller_obj_cache.go:2340	Avi cluster state is CLUSTER_UP_NO_HA
172023-02-04T11:43:53.176Z	INFO	cache/controller_obj_cache.go:2901	Setting cloud vType: CLOUD_VCENTER
182023-02-04T11:43:53.176Z	INFO	cache/controller_obj_cache.go:2904	Setting cloud uuid: cloud-ae84c777-ebf8-4b07-878b-880be6b201b5
192023-02-04T11:43:53.176Z	INFO	lib/lib.go:291	Setting AKOUser: ako-wdc-2-tkc-cluster-1 for Avi Objects
202023-02-04T11:43:53.177Z	INFO	cache/controller_obj_cache.go:2646	Skipping the check for SE group labels 
212023-02-04T11:43:53.332Z	INFO	cache/controller_obj_cache.go:3204	Setting VRF global found from network vds-tkc-frontend-vlan-1027
222023-02-04T11:43:53.332Z	INFO	record/event.go:282	Event(v1.ObjectReference{Kind:"Pod", Namespace:"avi-system", Name:"ako-0", UID:"4b0ee7bf-e5f5-4987-b226-7687c5759b4a", APIVersion:"v1", ResourceVersion:"41019", FieldPath:""}): type: 'Normal' reason: 'ValidatedUserInput' User input validation completed.
232023-02-04T11:43:53.336Z	INFO	lib/lib.go:230	Setting Disable Sync to: false
242023-02-04T11:43:53.338Z	INFO	k8s/ako_init.go:310	avi k8s configmap created

Looks good, let us try do deploy an application and expose it with Ingress

I have two demo applications, banana and apple. Yaml comes below. I deploy them

1andreasm@linuxvm01:~/ako$ k create ns fruit
2namespace/fruit created
3andreasm@linuxvm01:~/ako$ k apply -f apple.yaml -f banana.yaml 
4pod/apple-app created
5service/apple-service created
6pod/banana-app created
7service/banana-service created

yaml for banana and fruit

 1kind: Pod
 2apiVersion: v1
 3metadata:
 4  name: banana-app
 5  labels:
 6    app: banana
 7  namespace: fruit
 8spec:
 9  containers:
10    - name: banana-app
11      image: hashicorp/http-echo
12      args:
13        - "-text=banana"
14
15---
16
17kind: Service
18apiVersion: v1
19metadata:
20  name: banana-service
21  namespace: fruit
22spec:
23  selector:
24    app: banana
25  ports:
26    - port: 5678 # Default port for image
 1kind: Pod
 2apiVersion: v1
 3metadata:
 4  name: apple-app
 5  labels:
 6    app: apple
 7  namespace: fruit
 8spec:
 9  containers:
10    - name: apple-app
11      image: hashicorp/http-echo
12      args:
13        - "-text=apple"
14
15---
16
17kind: Service
18apiVersion: v1
19metadata:
20  name: apple-service
21  namespace: fruit
22spec:
23  selector:
24    app: apple
25  ports:
26    - port: 5678 # Default port for image

Then I apply the Ingress:

 1apiVersion: networking.k8s.io/v1
 2kind: Ingress
 3metadata:
 4  name: ingress-example
 5  namespace: fruit
 6#  annotations:
 7#    ako.vmware.com/enable-tls: "true"
 8
 9spec:
10  ingressClassName: avi-lb
11  rules:
12    - host: fruit-tkgs.you-have.your-domain.here
13      http:
14        paths:
15        - path: /apple
16          pathType: Prefix
17          backend:
18            service:
19              name: apple-service
20              port:
21                number: 5678
22        - path: /banana
23          pathType: Prefix
24          backend:
25            service:
26              name: banana-service
27              port:
28                number: 5678
1k apply -f ingress-example.yaml

You should more or less instantly notice a new virtual service in your Avi controller:

And let us check the ingress in k8s:

1andreasm@linuxvm01:~/ako$ k get ingress -n fruit 
2NAME              CLASS    HOSTS                                  ADDRESS       PORTS   AGE
3ingress-example   avi-lb   fruit-tkgs.you-have.your-domain.here   10.102.7.15   80      2m49s

There it is, with the actual VIP it gets from Avi.

Heres is the view of the application from the Dashboard view in Avi:

Also notice that the SE's now also places itself in the same network as the worker nodes, but still creates the VIP in the frontend-network.

Meaning our network diagram will now look like this: updated diagram with L7

Now, AKO comes with its own CRDs that one can work with. I will go through these in a separate post.