Antrea Network Policies


This post will go through the Antrea-native policy resources and how to use them to secure your Kubernetes environment combined with K8s network policies.

Abbreviations used in this article:

  • Container Network Interface = CNI
  • Antrea Cluster Network Policies = ACNP
  • Antrea Network Policies = ANP
  • Kubernetes Network Policies = K8s policies or KNP

When it comes to securing your K8s infrastructure it can be done in several layers in the infrastructure as a whole. This post will focus on the possibilities within the K8s cluster with features in the Antrea CNI. I will go through the Antrea-native policies (Antrea and K8s policies), with examples of when, how and where to use them. As Antrea-native policy resources can be used together with K8s network policies I will show that also.
This post will not cover the additional needs of security in your datacenter before reaching your K8s environment. I will cover this in a later post where I go through the use of NSX Distributed Firewall protecting your k8s clusters together with the security policies in Antrea.

Antrea-native policy resources - short introduction

Antrea comes with a comprehensive policy model. We have the Antrea Cluster Network Policies and Antrea Network Policies. The difference being between those two is that the ACNP applies to all objects on the cluster, where ANP is namespaced meaning its applies to objects within the namespace defined in the policy.
Antrea policies are tiered, meaning the rules will be following an order of precedence. This makes it very useful to divide the rules into the right categories, having different resources in the organization responsible for the security rules. Sec-ops will have their rules in the beginning setting the "ground" before the application owners can set their rules and finally some block all rules that rules out all that is left. Antrea-native policy resources is working together with K8s network policies where the latter is placed in the Application tier below the ANP and ACNP policies. Antrea comes with a set of default tiers as of installation of Antrea, but there is also possible to add custom tiers. Read more here. Here are the default tiers:

1Emergency   -> Tier name "emergency" with priority "50"
2SecurityOps -> Tier name "securityops" with priority "100"
3NetworkOps  -> Tier name "networkops" with priority "150"
4Platform    -> Tier name "platform" with priority "200"
5Application -> Tier name "application" with priority "250"
6Baseline    -> Tier name "baseline" with priority "253"

Take a look at the diagram below and imagine your first rules is placed at the first left tier and more rules in the different tiers all the way to the right:

Making use of the Antrea-native policy resources

In this section I will describe a demo environment with some namespaces and applications and then go through one way of using Antrea-native policy resources together with K8s network policies. In some of my Antrea-native policies I will use namespace selection based on the actual name of the namespace, but in others I will use selection based on labels. The first example below will make use of labels as selection criteria. I will explain why below. To read more on namespace selection:

Meet the demo environment

I will only use one K8s cluster in this example which is based on one master worker and two worker nodes.
I will create three "environments" by using namespaces and label them according to which "environment" they belong to. The three namespaces will be "test-app", "dev-app" and "prod-app". I will then add a label on each namespace with the label env=test, env=dev and env=prod accordingly. Within each namespace I will spin up two pods (an Ubuntu 16.04 and Ubuntu 20.04 pod). What I would like to achieve is that each namespace represents their own environment to simulate scenarios where we do have prod, dev and test environments, where none of the environments are allowed to talk to each other. And by using labels, I can create several namespaces and place them into the correct environment by just "tagging" them with the correct labels (e.g env=dev).

Now in the next section I will go through how I can isolate, and control those environments with Antrea-native policy resources.

Antrea Cluster Network Policies (ACNP)

The first thing I would like to do is to create some kind of basic separation between those environments/namespaces so they cant communicate with each other, and when that is done I can continue to create more granular application policies within each namespace or environment.
The first issue I meet is how to create as few rules as possible to just isolate what I know (the tree namespaces which are labeled with three different labels to create my "environments") without having to worry about additional namespaces being created and those getting access to the "environments". In this example I have already created three namespaces named "dev-app", "prod-app" and "test-app". I "tag" them in Kubernetes with their corresponding "env" labels: "dev", "prod" and "test". The reason I choose that approach is that I then can create several namespaces and choose which environment they belong to instead of doing the selection directly on the name of the namespace. I need to create an Antrea Cluster Network Policy as a "default" rule for each of my known environments so I can at a minimum guarantee that within each namespace or environment "intra-traffic" is allowed (traffic within the namespace or namespaces labeled with the same environment label). Meaning that when I do have a complete Antrea-native policy "framework" in place (with a blocking rule at the end taking care of all that is not specified) I can create new namespaces, but if they are not labeled correctly they will not be allowed to talk to any of my environments. This policy is applied at the SecurityOps tier:

 2kind: ClusterNetworkPolicy
 4  name: isolate-dev-env
 6    priority: 5
 7    tier: SecurityOps
 8    appliedTo:
 9      - namespaceSelector:
10          matchLabels:
11            env: dev
12    ingress:
13      - action: Drop
14        from:
15          - namespaceSelector:
16              matchExpressions:
17              - key: env
18                operator: NotIn
19                values:
20                  - dev
21      - action: Allow
22        from:
23          - namespaceSelector:
24              matchLabels:
25                env: dev
26    egress:
27      - action: Allow
28        to:
29          - namespaceSelector:
30              matchLabels:
31                env: dev

What I am also doing with this ACNP is saying that if you are member of a namespace with the label "env=dev" you are allowed to ingress the namespace Dev, but not if you are not ("operator: NotIn" in the ingress namespaceSelector).
Also note that I am allowing specifically an Action allow to the dev environment within the same policy, the reason being is that when I apply my block-all-else rule later on it will block intra traffic within the same environment if it is not specifically specified that it is allowed in this rule.
Now I just have to recreate this policy for my other two namespaces.
Also note that in the egress part I am only allowing traffic to namespace with the lavel "env=dev". That does not mean right now that I will only allow traffic to anything else, because I don't have any block rules in my cluster yet. Antrea-native policy resources works a bit different than K8s network policies which only supports creating allow policies. In Antrea one can specify both DROP and ALLOW on both INGRESS and EGRESS. I left this with purpose, because I later in will go ahead create a block all rule. Now lets demonstrate this rule:

Before applying ACNP namespace isolation rule:

1kubectl get pod -n dev-app -o wide
2NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
3ubuntu-16-04-7f876959c6-p5nxp   1/1     Running   0          9d   k8s-worker-01   <none>           <none>
4ubuntu-20-04-6fb66c64cb-9qg2p   1/1     Running   0          9d   k8s-worker-01   <none>           <none>
1kubectl get pod -n prod-app -o wide
2NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
3ubuntu-16-04-7f876959c6-sfdvf   1/1     Running   0          9d   k8s-worker-01   <none>           <none>
4ubuntu-20-04-6fb66c64cb-z528m   1/1     Running   0          9d   k8s-worker-01   <none>           <none>

Above I list out the pods with IP addresses in my two namespaces "dev-app" and "prod-app"
Now I enter bash of the Ubuntu20.04 pod in "dev-app" namespace and do a ping to the second pod in the same namespace and then ping another pod in the namespace Prod:

1kubectl exec -it -n dev-app ubuntu-20-04-6fb66c64cb-9qg2p bash
 1root@ubuntu-20-04-6fb66c64cb-9qg2p:/# ping
 2PING ( 56(84) bytes of data.
 364 bytes from icmp_seq=1 ttl=64 time=0.896 ms
 464 bytes from icmp_seq=2 ttl=64 time=0.520 ms
 564 bytes from icmp_seq=3 ttl=64 time=0.248 ms
 7root@ubuntu-20-04-6fb66c64cb-9qg2p:/# ping
 8PING ( 56(84) bytes of data.
 964 bytes from icmp_seq=1 ttl=64 time=1.03 ms
1064 bytes from icmp_seq=2 ttl=64 time=0.584 ms
1164 bytes from icmp_seq=3 ttl=64 time=0.213 ms

I have also written about Octant in one of my posts, in Octant there is an Antrea plugin which gives us some graphical features such as traceflow, which is also a powerful tool to showcase/troubleshoot security policies. Below is a screenshot from Octant before the rule is applied:

As you can see, this is allowed. Now I apply my "isolation" ACNP rules "prod", "dev" & "test". Also note; to list out the applied ACNP policies the command "kubetcl get acnp" can be used, without looking in a specific namespace as ACNP is clusterwide.

1kubectl apply -f created
3kubectl apply -f created
5kubectl get acnp
7isolate-dev-env    SecurityOps   5          1               1               19s
8isolate-prod-env   SecurityOps   6          1               1               25s

After they are applied I will try to do same as above:

2PING ( 56(84) bytes of data.
364 bytes from icmp_seq=1 ttl=64 time=3.28 ms
464 bytes from icmp_seq=2 ttl=64 time=0.473 ms
564 bytes from icmp_seq=3 ttl=64 time=0.190 ms
664 bytes from icmp_seq=4 ttl=64 time=0.204 ms
9PING ( 56(84) bytes of data.

Pinging within the same namespace works perfect, but to one of the other namespaces (here the Prod namespace) is not allowed. Works as intended.
Doing the same traceflow with Octant again:

So to recap, this is how it looks like now:

Now that I have created myself some isolated environments, I also need to allow some basic needs from the environments/namespaces to things such as DNS. So I will go ahead and create such a rule. Also have in mind that I haven't yet applied the last block-all-rule (so they can still reach those services as of now). I will make that rule applied when all the necessary rules are in place beforehand. In a greenfield environment those "baseline" rules would probably be applied as the first thing before the k8s cluster is taken into use.

Going down one tier to NetworkOps I will apply this Antrea Policy:

 2kind: ClusterNetworkPolicy
 4  name: allow-core-dns
 6    priority: 10
 7    tier: NetworkOps
 8    appliedTo:
 9      - namespaceSelector: {}
10    egress:
11      - action: Allow
12        to:
13          - namespaceSelector:
14              matchLabels:
15       kube-system
16        ports:
17          - protocol: TCP
18            port: 53
19          - protocol: UDP
20            port: 53

This policy is probably for some rather "wide" as it just does a "wildcard" selection of all namespaces available and gives them access to the backend kube-system (where the coredns pods are located) on protocol TCP and UDP port 53. But again, this post is just to showcase Antrea policies and how they can be used and to give some insights in general.
DNS allowed showed with Octant:

Octant Traceflow

For now I am finished with the Cluster Policies and will head over to the Network Policies. This is the ACNP policies applied so far:

1kubectl get acnp
3allow-core-dns     NetworkOps    10         3               3               19h
4isolate-dev-env    SecurityOps   5          1               1               22h
5isolate-prod-env   SecurityOps   6          1               1               22h
6isolate-test-env   SecurityOps   7          1               1               19h

Antrea Network Policies (ANP)

Antrea Network Policies are namespaced. So one of the use cases for ANP could be to create rules specific for the services running in the namespace. It could be allowing ingress on certain selections (e.g label/frontend) which runs the application I want to expose or which makes sense for clients to talk to which is the frontend part of the application. Everything else is backend services which not necessary to expose to clients, but on the other hand it could be that those services needs access to other backend services or services in other namespaces. So with ANP one can create ingress/egress policies by using the different selection options defining what is allowed in and out of the namespace.

Before I continue I have now applied my ACNP block-rule in the "Baseline" tier which just blocks all else to make sense of the examples used here in this section. Below is the policy (Note that I have excluded some namespaces in this rule) :

 2kind: ClusterNetworkPolicy
 4  name: block-all-whitelist
 6    priority: 1000
 7    tier: baseline
 8    appliedTo:
 9      - namespaceSelector:
10          matchExpressions:
11          - key: ns
12            operator: NotIn
13            values:
14              - kube-system
15              - monitoring
16    ingress:
17      - action: Drop
18        from:
19          - namespaceSelector: {}
20          - ipBlock:
21              cidr:
22    egress:
23      - action: Drop
24        to:
25          - namespaceSelector: {}
26          - ipBlock:
27              cidr:

Antrea Network Policies Egress rule

Now that I have applied my "whitelist" rule I must by now have all my necessary rules in place, otherwise things will stop working (such as access to DNS). I will now apply a policy which is "needed" by the "Prod" environment, which is access to SSH on a remote server. So the policy below is allowing Egress on TCP port 22 to this specific remote SSH server. Lets us apply this policy and test how this works out:

 2kind: NetworkPolicy
 4  name: allow-prod-env-ssh
 5  namespace: prod-app
 7    priority: 8
 8    tier: application
 9    appliedTo:
10      - podSelector: {}
11    egress:
12      - action: Allow
13        to:
14          - namespaceSelector:
15              matchLabels:
16       prod-app
17          - ipBlock:
18              cidr:
19        ports:
20          - protocol: TCP
21            port: 22

Just some sanity check before applying the above policy, I am still able to reach all pods within the same namespace due to my "isolation ACNP rules" even though I have my block all rule applied.
But I am not allowed to reach anything outside except what is stated in my DNS rule. If I try to reach my remote SSH server from my "Prod" namespace I am not allowed. To illustrate this I have entered "remoted" myself into bash on one of my pods in the Prod namespace and trying to ssh the remote server, below is the current result:

1root@ubuntu-20-04-6fb66c64cb-z528m:/# ssh andreasm@
2ssh: connect to host port 22: Connection timed out

Ok, fine. What does my traceflow say about this also:

Nope cant do it also says Traceflow. Great, everything works out as planned. Now I must apply my policy to allow this.

1root@ubuntu-20-04-6fb66c64cb-z528m:/# ssh andreasm@
2andreasm@'s password:

Now I can finally reach my remote SSH server. To confirm again, lets check with Octant:

Thank you very much, that was very kind of you.

To summarize so far what we have done. We have applied the ACNP rules/policies to create environment/namespace isolation

2allow-core-dns        NetworkOps    10         3               3               21h
3block-all-whitelist   baseline      20         2               2               25m
4isolate-dev-env       SecurityOps   5          1               1               24h
5isolate-prod-env      SecurityOps   6          1               1               31m
6isolate-test-env      SecurityOps   7          1               1               21h

And we have applied a rule to allow some basic "needs" such as DNS with the rule allow-core-dns. And the "block-all-whitelist" policy as a "catch all rule" to block everything not specified in the tiers.
And then we have applied a more application/namespace specific policy with Antrea Network Policy to allow Prod to egress on port 22/TCP. But I have not specified any ingress rules allow access to any services in the namespace Prod coming from outside the namespace. So it is a very lonely/isolated environment for the moment. This is how it looks like now:

In the next example I will create an ingress rule to another application that needs to be accessed from the outside.

Antrea Network Policies Ingress rule

To make this section a bit more "understanding" I will use another application as example to easier illustrate the purpose. The example I will be using is a demo application I have been using for several years - Yelb link

This application contains of four pods and looks like this:

Yelb diagram

I already have the application up and running in my environment. But as this application is a bit more complex and contains a frontend which is useless if not exposed or reachable I am exposing this frontend with NSX Advanced Load Balancer. This makes it very easy for me to define the ingress rule as it means I only have to allow the load balancers IPs in my egress rule and not all potential IPs. The load balancers IP's is something I know. Some explanation around the load balancer IP's in my environment is that they are spun up on demand and just pick an IP from a pre-defined IP pool, so instead of pinning the ingress rule to the current IP they have I am bit wide and allow the IP range that is defined. Remember that this is a demo environment and does not represent a production environment. Lets take a look at the policy:

 2kind: NetworkPolicy
 4  name: allow-yelb-frontend
 5  namespace: yelb
 7    priority: 5
 8    tier: application
 9    appliedTo:
10      - podSelector:
11          matchLabels:
12            app: yelb-ui
13    ingress:
14      - action: Allow
15        from:
16          - ipBlock:
17              cidr:
18        ports:
19          - protocol: TCP
20            port: 80
21            endPort: 80
22        name: AllowInYelbFrontend
23        enableLogging: false
24    egress:
25      - action: Allow
26        to:
27          - ipBlock:
28              cidr:
29        name: AllowOutYelbFrontend
30        enableLogging: false

The CIDR in the rule above is the range my load balancers is "living" in. So instead to narrow it down too much in this demo I just allow the range meaning I dont have to worry too much if they are getting new IP's within this range making my application inaccessible. When I apply this rule it will be placed in the tier "Application" (See one of the first diagrams in the beginning of this post) with a priority of 5. The basic policies for this application is already in place such as DNS and intra-communication (allowed to talk within the same namespace/environment which in this example is yelb-app/test).

Now lets see how it is before applying the rule from the perspective of the NSX Advanced Load Balancer which is being asked to expose the frontend of the application:

From NSX Advanced Load Balancer GUI / application showing pool is down

As one can see from the above screenshot, the Service Engines (the actual load balancers) are up and running but the application yelb-ui is down because the pool is unreachable. The pool here is the actual pod containing the frontend part of the Yelb app. So I need to apply the Antrea Network Policy to allow the Service Engines to talk to my pod. If I try to access the frontend via the load balancer VIP its also inaccessible:

Lets just apply the rule:

1kubectl apply -f yelb.frontend.allow.yaml created
3kubectl get anp -n yelb
5allow-yelb-frontend   application   5          1               1               11s

And now check the NSX Advanced Load Balancer status page and try to access the application through the VIP:

NSX ALB is showing green

Well that looks promising, green is a wonderful colour in IT.
And the application is available:

Yelb UI frontend

The rule above only gives the NSX Advanced Load Balancers access to the frontend pod on port 80 of the application Yelb. All the other pods are protected by the "environment" block rule and the default block rule. There is one catch though, we dont have any rules protecting traffic between the pods. Lets say the frontend pod (which is exposed to the outside world) is compromised, there is no rule stopping any traffic coming from this pod to the others within the same namespace/and or environment. That is something we should apply.

Microsegmenting the application

What we shall do now is to make sure that the pods that make up the application Yelb is only allowed to talk to each other on the necessary ports and nothing else. Meaning we create a policy that does a selection of the pods and apply specific rules for each pod/service within the application, if one refer to the diagram above over the Yelb application one can also see that there is no need for the fronted pod to be allowed to talk to the redis or db pod at all so that should be completely blocked.
I will go ahead and apply a rule that does all the selection for me, and only allow what is needed for the application to work. The policy I will make use of here is K8s Native Network Policy
Here is the rule:

 2kind: NetworkPolicy
 4  name: yelb-cache
 5  namespace: yelb
 7  podSelector:
 8    matchLabels:
 9      tier: cache
10  ingress:
11  - from:
12    - podSelector:
13        matchLabels:
14          tier: middletier
15    - namespaceSelector:
16        matchLabels:
17          tier: middletier
18    ports:
19    - protocol: TCP
20      port: 6379
21  policyTypes:
22  - Ingress
25kind: NetworkPolicy
27  name: yelb-backend
28  namespace: yelb
30  podSelector:
31    matchLabels:
32      tier: backenddb
33  ingress:
34  - from:
35    - podSelector:
36        matchLabels:
37          tier: middletier
38    - namespaceSelector:
39        matchLabels:
40          tier: middletier
41    ports:
42    - protocol: TCP
43      port: 5432
44  policyTypes:
45  - Ingress
48kind: NetworkPolicy
50  name: yelb-middletier
51  namespace: yelb
53  podSelector:
54    matchLabels:
55      tier: middletier
56  ingress:
57  - from:
58    - podSelector: {}
59    - namespaceSelector:
60        matchLabels:
61          tier: frontend
62    ports:
63    - protocol: TCP
64      port: 4567
65  policyTypes:
66  - Ingress
69kind: NetworkPolicy
71  name: yelb-frontend
72  namespace: yelb
74  podSelector:
75    matchLabels:
76      tier: frontend
77  ingress:
78  - from:
79    ports:
80    - protocol: TCP
81      port: 80
82  egress:
83  - to:
84    ports:
85    - protocol: TCP
86      port: 30567
87  policyTypes:
88  - Ingress
89  - Egress

As I have already illustrated above I will not go through showing that the pods can talk to each other on all kinds of port, as they can because they do not have any restriction within the same namespace/environment. What I will go through though is how the above policy affects my application.
The rule applied:

 1kubectl apply -f k8snp_yelb_policy.yaml created created created created
 7kubectl get -n yelb
 8NAME              POD-SELECTOR      AGE
 9yelb-backend      tier=backenddb    80s
10yelb-cache        tier=cache        80s
11yelb-frontend     tier=frontend     80s
12yelb-middletier   tier=middletier   80s

So to illustrate I will paste a diagram with the rules applied, and go ahead an see if I am allowed and not allowed to reach pods on ports not specified.

Yelb diagram with policies

The first thing I will try is to see if the frontend pod can reach the appserver on the specified port 4567:

Octant Antrea Traceflow

And the result is in:

Now, what if I just change the port to something else, say DNS 53... Will it succeed?