Kubernetes NetworkPolicy Rules We Can Actually Prove

Build a locked-down namespace, then test every allowed connection ourselves.

The orders-api namespace was accepting traffic from a forgotten batch job at 08:46, which was annoying because we had already written “internal only” in three tickets. Kubernetes had done exactly what we told it to do: nothing. Pods can generally talk to other pods unless the network plugin says otherwise.

We fixed the small version of that problem first. By the end of this walkthrough, we’ll have an orders-api namespace where:

• the API accepts port 8080 traffic only from its frontend;
• the API can reach PostgreSQL on 5432;
• the API can resolve DNS;
• random pods cannot connect to the API or send arbitrary outbound traffic.

This uses standard NetworkPolicy objects. We tested the same manifests on Kubernetes 1.30 with Cilium 1.16. Calico works too, but do not assume every cluster’s CNI enforces policies. Kubernetes will happily accept a NetworkPolicy object that changes precisely nothing. A very Kubernetes outcome.

Check Whether The Cluster Enforces NetworkPolicy

First, point kubectl at a cluster we can safely alter.

# Confirm the current context before creating anything.
kubectl config current-context

# Confirm the Kubernetes server version.
kubectl version

# See which CNI components are running.
kubectl get pods -n kube-system -o wide

On our clusters, Cilium pods show up as cilium-xxxxx in kube-system. If we’re using Calico, we expect calico-node-xxxxx. The exact pod names are less interesting than whether the plugin supports policy enforcement.

Check the plugin documentation rather than trusting a chart value from six months ago. The relevant references are Cilium’s Kubernetes NetworkPolicy docs and Calico’s policy documentation.

For a disposable local cluster, we use kind with Cilium installed separately. This is optional if there’s already a policy-capable cluster available.

# Create a local Kubernetes cluster for the exercise.
kind create cluster --name networkpolicy-lab

# Install the Cilium CLI if it is not already installed.
# macOS example:
brew install cilium

# Install Cilium and wait until its agents are ready.
cilium install
cilium status --wait

# Point kubectl at the new kind cluster.
kubectl config use-context kind-networkpolicy-lab

Don’t move past this step on faith. A policy test on a cluster with no enforcing CNI gives us a false sense of safety, which is worse than no test because somebody will cite it during an incident review.

Create The Three Workloads We Need

We need an API, a frontend client, and a PostgreSQL-shaped target. The database is only postgres:16-alpine; we are testing network paths, not writing an orders system before lunch.

Save this as 00-workloads.yaml.

# 00-workloads.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: orders-api
  labels:
    team: commerce
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: orders-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
        role: backend
    spec:
      containers:
        - name: api
          image: hashicorp/http-echo:1.0
          args:
            - "-listen=:8080"
            - "-text=orders-api is alive"
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: orders-api
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  namespace: orders-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
        role: frontend
    spec:
      containers:
        - name: frontend
          image: curlimages/curl:8.10.1
          command: ["sleep", "3600"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: orders-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
        role: database
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_PASSWORD
              value: not-a-real-password
          ports:
            - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: orders-api
spec:
  selector:
    app: postgres
  ports:
    - name: postgresql
      port: 5432
      targetPort: 5432

Apply it and wait for the deployments.

# Create the namespace, deployments, and Services.
kubectl apply -f 00-workloads.yaml

# Wait for all three deployments to become available.
kubectl rollout status deployment/api -n orders-api
kubectl rollout status deployment/frontend -n orders-api
kubectl rollout status deployment/postgres -n orders-api

# Keep this output around; labels drive the policy selectors later.
kubectl get pods -n orders-api --show-labels

We deliberately use workload labels such as app: frontend rather than pod names. Pod names change. Labels are the contract here, so they deserve the same review attention as an IAM role binding.

Prove The Namespace Starts Wide Open

Before adding any policy, both the frontend and an unapproved pod should reach the API. Create the unapproved pod in a separate namespace so we prove that cross-namespace traffic is also open.

# Create a namespace representing an unrelated workload.
kubectl create namespace reporting

# Start a temporary curl pod and leave it alive for testing.
kubectl run reporter \
  --namespace reporting \
  --image=curlimages/curl:8.10.1 \
  --command -- sleep 3600

# Wait until the test pod has an IP address.
kubectl wait \
  --namespace reporting \
  --for=condition=Ready pod/reporter \
  --timeout=90s

# Confirm the approved frontend can call the API.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Confirm the unrelated reporter can also call it before policy exists.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

Both commands should print orders-api is alive. If the second one already fails, stop and find out why. Perhaps there is an existing global policy, a service mesh rule, or someone has been more diligent than we expected. Copying a baseline test from a blog into a cluster with unknown controls is how we manufacture confusing evidence.

Deny All Inbound Traffic To The API

NetworkPolicy becomes active for a pod in a direction when a policy selects that pod and includes that direction. This policy selects API pods and says all ingress is denied unless another policy permits it.

Save this as 01-api-default-deny.yaml.

# 01-api-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-deny-all-ingress
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress

Apply it, then repeat both checks.

# Turn on ingress isolation for API pods.
kubectl apply -f 01-api-default-deny.yaml

# The frontend should now fail with a connection timeout.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# The unrelated namespace should fail too.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

The curl exit code should be non-zero, usually 28 for a timeout. A rejected TCP connection can also happen depending on the CNI. What we should not see is the response body.

Keep the timeout short in these checks. A default curl timeout during a failed policy test feels like waiting for a kettle that has been unplugged.

This is the shortest section because the object itself is mercifully small. The next policy puts back exactly one route.

Allow Only Frontend Requests On Port 8080

The API needs inbound HTTP from frontend pods in the same namespace. We specify both the namespace selector and the pod selector. Omitting namespaceSelector is easy here because the policy lives in orders-api, but we prefer being explicit in security controls. It makes a later copy into another namespace less surprising.

# 02-api-allow-frontend.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-frontend-http
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders-api
          podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

Apply it and run the same calls again.

# Permit the frontend-to-API path only.
kubectl apply -f 02-api-allow-frontend.yaml

# This should return the API response now.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# This must remain blocked.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Inspect the effective intent in Kubernetes, not just the local file.
kubectl describe networkpolicy -n orders-api api-allow-frontend-http

Policies are additive. The deny policy does not override the allow policy; it establishes isolation, and this policy adds a permitted ingress path. We still see people expect “deny all” to win like a firewall rule at the bottom of a list. Kubernetes NetworkPolicy does not work that way, and frankly I prefer the additive model once everyone has been burned by it once.

Restrict API Egress To PostgreSQL And DNS

Now we isolate egress from the API. This catches the overlooked half of many namespace policies: an exploited API that can call every service and every public endpoint is still having a fairly productive day.

First, find the DNS service IP and labels in this cluster. CoreDNS commonly has k8s-app: kube-dns, but check rather than inventing labels at runtime.

# Find the DNS service and its ClusterIP.
kubectl get service -n kube-system kube-dns

# Inspect CoreDNS pod labels for the policy selector.
kubectl get pods -n kube-system -l k8s-app=kube-dns --show-labels

Create the egress policy. The PostgreSQL rule permits TCP/5432 to pods labeled app: postgres in orders-api. The DNS rule permits UDP and TCP port 53 to CoreDNS pods in kube-system.

# 03-api-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow-postgres-and-dns
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders-api
          podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

# Enable egress isolation and allow only database plus DNS traffic.
kubectl apply -f 03-api-egress.yaml

# DNS resolution should still work from the API pod.
kubectl exec -n orders-api deploy/api -- \
  wget -qO- http://postgres.orders-api.svc.cluster.local:5432 || true

# TCP connectivity to PostgreSQL should succeed, even though HTTP is nonsense there.
kubectl exec -n orders-api deploy/api -- \
  sh -c 'nc -zvw 3 postgres.orders-api.svc.cluster.local 5432'

# Arbitrary outbound HTTPS should now time out.
kubectl exec -n orders-api deploy/api -- \
  wget -T 3 -qO- https://example.com

The first command may emit PostgreSQL protocol noise because wget is speaking HTTP to a database. That is expected and mildly ugly. The nc command is the useful test; it should report a successful connection.

We have not tried this exact label-based DNS rule past 40 nodes with NodeLocal DNSCache enabled. That setup changes the destination path, and we still have an open argument about whether to standardise it this quarter. If NodeLocal DNSCache is in use, test it from a real pod before declaring DNS allowed.

Add A Default Deny For New Pods

The API is isolated, but a new pod added to orders-api would still have open ingress and egress unless its own policy selects it. We add namespace-wide default denial so new workloads begin closed.

This is our preference for service namespaces. It forces each deployment PR to state its network needs. It also creates more policy files, which is a fine trade when the alternative is guessing why a maintenance job can reach payroll.

# 04-namespace-default-deny.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: namespace-default-deny
  namespace: orders-api
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

# Apply isolation to every current and future pod in orders-api.
kubectl apply -f 04-namespace-default-deny.yaml

# The frontend now has no egress policy, so its API call should fail.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# List every policy now affecting the namespace.
kubectl get networkpolicy -n orders-api

That failed frontend call is correct. We broke it deliberately, in daylight, while we have the terminal and the attention span to repair it.

Restore The Frontend’s Required Egress

The frontend needs DNS and TCP/8080 to the API. Add those two paths, then rerun the test. We use one policy per workload rather than one huge namespace policy because code review stays readable after the fourth service arrives.

# 05-frontend-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-allow-api-and-dns
  namespace: orders-api
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: orders-api
          podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 8080
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

# Restore only the frontend's required outbound paths.
kubectl apply -f 05-frontend-egress.yaml

# Approved frontend traffic works again.
kubectl exec -n orders-api deploy/frontend -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Cross-namespace traffic remains blocked.
kubectl exec -n reporting pod/reporter -- \
  curl --connect-timeout 3 -sS http://api.orders-api.svc.cluster.local:8080

# Show all objects we would commit with the application manifests.
kubectl get networkpolicy -n orders-api

At this point, we would commit the five YAML files beside the Helm chart or Kustomize overlay that owns orders-api. Keeping policies in a separate security repository sounds tidy until an application team changes its port and the policy change waits in another queue. We tried that arrangement in 2024. It produced a lot of polite Slack messages and one very long release.

When This Goes Sideways

If every connection still works after applying api-deny-all-ingress, the CNI probably is not enforcing NetworkPolicy. Check its DaemonSet, its logs, and any managed-cluster setting that enables policy support. kubectl get networkpolicy only proves the API server stored YAML.

If DNS fails after egress isolation, inspect the actual CoreDNS labels and check whether NodeLocal DNSCache is enabled:

# Check CoreDNS labels and look for NodeLocal DNSCache.
kubectl get pods -n kube-system --show-labels | grep -E 'coredns|node-local-dns'

# Inspect DNS configuration from the affected pod.
kubectl exec -n orders-api deploy/api -- cat /etc/resolv.conf

# Review policy events and the API pod's recent state.
kubectl describe pod -n orders-api -l app=api
kubectl get events -n orders-api --sort-by=.lastTimestamp

If a policy seems correct but traffic still fails, test by pod IP as well as Service DNS. Service handling varies by CNI mode, kube-proxy replacement, and whether traffic is observed before or after translation. We do not guess which one applies; we record the cluster version and CNI mode in the incident ticket.

# Get the PostgreSQL pod IP for a direct connectivity comparison.
POSTGRES_IP=$(kubectl get pod -n orders-api -l app=postgres \
  -o jsonpath='{.items[0].status.podIP}')

# Test direct TCP traffic from the API pod.
kubectl exec -n orders-api deploy/api -- \
  sh -c "nc -zvw 3 ${POSTGRES_IP} 5432"

Finally, remove the lab when it has served its purpose. Leave it around and somebody will point a real test at postgres because the service name looked convenient.

# Remove the exercise namespaces and all namespaced resources.
kubectl delete namespace orders-api reporting

# Delete the local kind cluster if we created one.
kind delete cluster --name networkpolicy-lab

Tags: devopsKubernetes