K8s Ingress with NSX Advanced Load Balancer

Abbreviations used in this article:

• NSX Advanced Load Balancer = NSX-ALB
• K8s = Kubernetes (8 letters between the K and s in Kubernetes)
• SSL = Secure Sockets Layer
• AKO = Avi Kubernetes Operator (AVI now a VMware product called NSX Advanced Load Balancer)

In one of my previous posts I wrote about how to install and configure AKO (Avi Kubernetes Operator) to use as Service type LoadBalancer.

This post will try to cover the basics of how to use NSX Advanced LoadBalancer by using AKO to handle our Ingress requests (ingress-controller).
For more information on Ingress in Kubernetes

An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.

Ingress Load Balancer Kubernetes Definition

Within Kubernetes or K8s, a collection of routing rules that control how Kubernetes cluster services are accessed by external users is called ingress. Managing ingress in Kubernetes can take one of several approaches.

An application can be exposed to external users via a Kubernetes ingress resource; a Kubernetes NodePort service which exposes the application on a port across each node; or using an ingress load balancer for Kubernetes that points to a service in your cluster.

An external load balancer routes external traffic to a Kubernetes service in your cluster and is associated with a specific IP address. Its precise implementation is controlled by which service types the cloud provider supports. Kubernetes deployments on bare metal may require custom load balancer implementations.

However, properly supported ingress load balancing for Kubernetes is the simplest, more secure way to route traffic. link

Getting AKO ready

While this post assumes AKO is already in place and working in your k8s clusters I will get straight to the parts that involve Ingress. If not head over here to read the official docs how to install Avi Kubernetes Operator (AKO). To verify that you AKO is ready to handle Ingress request, type in this and notice the output:

1$ kubectl get ingressclasses.networking.k8s.io 
2NAME     CONTROLLER              PARAMETERS   AGE
3avi-lb   ako.vmware.com/avi-lb   <none>       50d

Default secret for TLS Ingress

As AKO expects all ingresses with TLS termination to have a key and certificate specified, there is a couple of ways this can be done. We can go with a "pr service", meaning a dedicated set of keys and certs pr service or a default/common set of keys and certificates that AKO can use if nothing else is specified. To apply the common approach, one common key and certificate for one or more applications we need to add a secret for the AKO. Prepare your router-default.yaml definition file like this (the official docs wants you to put in your cert as is, that does not work so you need to base64 encode both keys and certs and paste in below):

 1apiVersion: v1
 2kind: Secret
 3metadata:
 4  name: router-certs-default
 5  namespace: avi-system
 6type: kubernetes.io/tls
 7data:
 8  tls.key: "base64 encoded"
 9  tls.crt: "base64 encoded"
10  alt.key: "base64 encoded"
11  alt.crt: "base64 encoded"
12  

To base64 encode your keys and certs this can be done like this:

If you have the keys and certs in a file, from whatever linux terminal type in:

1cat cert.pem | base64 -w 0 
2cat key.pem | base64 -w 0

Then paste into the above yaml accordingly (tls.key:key, tls.crt:crt) If you have both ECDSA and RSA certs use the alt.key and alt.crt to apply both. As soon as everything is pasted, apply the yaml file kubectl apply -f router-defaults.yaml

Apply your ingress service

To create an ingress service you need to define this in yaml. An example below:

 1apiVersion: networking.k8s.io/v1
 2kind: Ingress
 3metadata:
 4  name: "NameOfIngress-Service"
 5  namespace: "Namespaceofwhereyour-service-resides"
 6  labels:
 7    app: "ifyouwant"
 8  annotations:
 9    ako.vmware.com/enable-tls: "true" #"Indicates to Avi that you want to use TLS"
10spec:
11  ingressClassName: avi-lb #"The default class name for AVI"
12  rules:
13    - host: "FQDN"
14      http:
15        paths:
16        - pathType: Prefix
17          path: /
18          backend:
19            service:
20              name: "which-service-to-point-to"
21              port:
22                number: 80

Hostrules and HTTPrules

As I mentioned earlier, you can also define specific rules pr server such as certificates. Here we can use Hostrules and HTTPrules to further adjust granular settings pr service. One Hostrule example below:

 1apiVersion: ako.vmware.com/v1alpha1
 2kind: HostRule
 3metadata:
 4  name: name-of-your-rule
 5  namespace: namespace-of-your-service
 6spec:
 7  virtualhost:
 8    fqdn: must-match-hostname-above # mandatory
 9    fqdnType: Exact
10    enableVirtualHost: true
11    tls: # optional
12      sslKeyCertificate:
13        name: "name-of-certificate" # This must be already defined in your AVI controller
14        type: ref
15        alternateCertificate:
16          name: "name-of-alternate-cert" # This must be already defined in your AVI controller
17          type: ref
18      sslProfile: System-Standard-PFS
19      termination: edge

To get all features available head over to the official docs site here

Hostrule example from the official docs:

 1apiVersion: ako.vmware.com/v1alpha1
 2kind: HostRule
 3metadata:
 4  name: my-host-rule
 5  namespace: red
 6spec:
 7  virtualhost:
 8    fqdn: foo.region1.com # mandatory
 9    fqdnType: Exact
10    enableVirtualHost: true
11    tls: # optional
12      sslKeyCertificate:
13        name: avi-ssl-key-cert
14        type: ref
15        alternateCertificate:
16          name: avi-ssl-key-cert2
17          type: ref
18      sslProfile: avi-ssl-profile
19      termination: edge
20    gslb:
21      fqdn: foo.com
22      includeAliases: false
23    httpPolicy: 
24      policySets:
25      - avi-secure-policy-ref
26      overwrite: false
27    datascripts:
28    - avi-datascript-redirect-app1
29    wafPolicy: avi-waf-policy
30    applicationProfile: avi-app-ref
31    analyticsProfile: avi-analytics-ref
32    errorPageProfile: avi-errorpage-ref
33    analyticsPolicy: # optional
34      fullClientLogs:
35        enabled: true
36        throttle: HIGH
37      logAllHeaders: true
38    tcpSettings:
39      listeners:
40      - port: 8081
41      - port: 6443
42        enableSSL: true
43      loadBalancerIP: 10.10.10.1
44    aliases: # optional
45    -  bar.com
46    -  baz.com

Httprule example from the official docs:

 1apiVersion: ako.vmware.com/v1alpha1
 2kind: HTTPRule
 3metadata:
 4   name: my-http-rule
 5   namespace: purple-l7
 6spec:
 7  fqdn: foo.avi.internal
 8  paths:
 9  - target: /foo
10    healthMonitors:
11    - my-health-monitor-1
12    - my-health-monitor-2
13    loadBalancerPolicy:
14      algorithm: LB_ALGORITHM_CONSISTENT_HASH
15      hash: LB_ALGORITHM_CONSISTENT_HASH_SOURCE_IP_ADDRESS
16    tls: ## This is a re-encrypt to pool
17      type: reencrypt # Mandatory [re-encrypt]
18      sslProfile: avi-ssl-profile
19      destinationCA:  |-
20        -----BEGIN CERTIFICATE-----
21        [...]
22        -----END CERTIFICATE-----