Q: What MetalLB mode causes this, and what is the failover behavior of each mode?

MetalLB has two modes: Layer 2 (ARP/NDP) and BGP. In Layer 2 mode, one node assumes the IP using ARP. When that node fails, ARP failover depends on clients refreshing their ARP cache, which typically takes 2-5 minutes. This causes the 5-minute blackout. In BGP mode, all nodes advertising the Service IP withdraw the route immediately upon failure, and BGP convergence (with BFD) redirects traffic in under 1 second. The fix is to migrate from Layer 2 to BGP mode for production Services.

Q: What five things do you check, starting from the most likely cause?

1. Service name and namespace: verify with kubectl get svc. 2. CoreDNS is running: kubectl get pods -n kube-system -l k8s-app=kube-dns. 3. CoreDNS Corefile: Check if the kubernetes plugin is configured correctly. 4. Pod resolv.conf: should show nameserver 10.96.0.10 and search domains. 5. NetworkPolicy blocking UDP 53: A default-deny policy may block DNS queries.

Question 1

What are the three most likely causes, and how do you debug each?

Accepted Answer

First, NetworkPolicy blocking egress from production or ingress to monitoring. I check kubectl get networkpolicy -n production and kubectl get networkpolicy -n monitoring. If a default-deny policy exists in monitoring, I verify that an explicit ingress rule allows traffic from the production namespace label. Second, DNS resolution failure. The Pod may be trying to reach service-name without the namespace suffix. In Kubernetes, cross-namespace Service access requires service-name.monitoring.svc.cluster.local. If the Pod uses just service-name, it resolves to the Service in its own namespace (if any) or fails. I debug with kubectl exec -it pod -- nslookup service-name.monitoring.svc.cluster.local. Third, Service port mismatch. The Service targets port 8080 but the client connects to port 80. I verify with kubectl get svc -n monitoring and check the targetPort vs port fields.

Question 2

What Calico configuration causes this latency spike, and what is the fix?

Accepted Answer

The latency spike is likely caused by Calico in IPIP (IP-in-IP) mode with cross-subnet IPIP enabled. When nodes are in different L2 subnets (even in the same DC), Calico encapsulates all Pod traffic in IPIP tunnels, adding overhead and often hairpinning through the default gateway. With 50 new nodes, the probability of cross-subnet traffic increased. The fix is to migrate to VXLAN encapsulation or BGP mode with route reflection. VXLAN has better performance than IPIP and supports VXLAN offload on modern NICs. The best fix is Calico BGP mode without encapsulation: each node advertises Pod CIDRs via BGP to the ToR switches, enabling native L3 routing. This eliminates encapsulation overhead entirely and restores sub-1ms latency.

Question 3

What three misconfigurations between the Ingress, Service, and backend Pods cause 502 errors?

Accepted Answer

First, Ingress backend protocol mismatch. If the Ingress terminates TLS and forwards to the Service as HTTPS, but the backend Pods only speak HTTP, the Ingress cannot establish a TLS connection with the backend. The fix is to add the annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTP" or configure the Service to use port 80. Second, readiness probe failure. If the backend Pod readiness probe fails after the HTTPS change, the Pod is removed from the Service endpoints. I verify with kubectl get endpoints and kubectl describe pod. Third, Ingress SSL/TLS certificate issue. If the Ingress uses a self-signed cert and the client rejects it, some Ingress controllers return 502 instead of a proper TLS error. I check the Ingress TLS secret with kubectl get secret tls-secret -o yaml and verify the cert chain is complete.

Question 4

What common NetworkPolicy misconception causes this, and what is the correct policy spec?

Accepted Answer

The most common misconception is using namespaceSelector with name: frontend instead of matching the namespace labels. Kubernetes NetworkPolicy namespaceSelector matches based on namespace labels, not the namespace name. If the frontend namespace does not have a label name: frontend, the selector does not match anything. I verify namespace labels with kubectl get namespace frontend --show-labels. The correct policy either uses a label that exists on the namespace or explicitly selects the namespace by creating a label for it. Another common issue is default-deny egress on the frontend Pods. NetworkPolicy is directional — both ingress and egress must be permitted.

Question 5

What eBPF feature replaces kube-proxy, and what upgrade step is often missed?

Accepted Answer

Cilium replaces kube-proxy with eBPF-based load balancing using a BPF map that stores Service backends. Instead of iptables rules, Cilium uses eBPF programs attached to tc hooks to rewrite packet destinations directly in the kernel. The common missed upgrade step is BPF map incompatibility between Cilium versions. If the old BPF maps are not removed during upgrade, the new Cilium agent cannot parse them, causing stale or missing Service entries. The fix is to run cilium bpf lb list and verify the Service backends are correct. If not, I delete the old BPF maps and restart Cilium agents.

Question 6

How does headless Service DNS differ from ClusterIP DNS, and what is the correct DNS name format for StatefulSet Pods?

Accepted Answer

A ClusterIP Service returns a single A record with the virtual IP. A headless Service returns A records for each Pod IP directly, enabling direct Pod-to-Pod communication without load balancing. For a StatefulSet, CoreDNS creates SRV records and individual A records for each Pod. The correct DNS format is: pod-name.service-name.namespace.svc.cluster.local. For example, pod-2.my-service.production.svc.cluster.local. If Pod-1 cannot resolve this, the causes are: CoreDNS not running, StatefulSet serviceName mismatch, or ndots:5 default in resolv.conf.

Question 7

What Istio feature allows gradual mTLS migration without breaking legacy apps, and how do you implement it?

Accepted Answer

The feature is PeerAuthentication with PERMISSIVE mode and mTLS migration using DestinationRules. Istio supports three mTLS modes: PERMISSIVE (accepts both plaintext and mTLS), STRICT (mTLS only), and DISABLE (plaintext only). For gradual migration, I set the mesh-wide PeerAuthentication to PERMISSIVE. This allows legacy apps to continue working while new apps use mTLS. Then I create DestinationRules for specific services to require mTLS. Once all clients are confirmed to use mTLS, I switch the namespace-level PeerAuthentication to STRICT. For legacy apps that cannot be upgraded, I deploy an Istio sidecar next to them and use EGRESS gateway or ServiceEntry to handle mTLS on their behalf.

Question 8

What node-level networking difference causes this, and what is the security risk of hostNetwork Pods?

Accepted Answer

The difference is that hostNetwork Pods share the host network namespace, using the node IP and routing table directly. Standard Pods use the CNI overlay network, which may have a different default route or NAT configuration. If the CNI overlay does not have a default route or if the overlay subnet is not NATd by the node, standard Pods cannot reach the internet. The security risk of hostNetwork is severe: The Pod bypasses NetworkPolicy enforcement because NetworkPolicy operates at the CNI level. The Pod can bind to privileged ports (<1024) on the node IP. The Pod can access all host network interfaces, including the management network.

Question 9

What MetalLB mode causes this, and what is the failover behavior of each mode?

Accepted Answer

MetalLB has two modes: Layer 2 (ARP/NDP) and BGP. In Layer 2 mode, one node assumes the IP using ARP. When that node fails, ARP failover depends on clients refreshing their ARP cache, which typically takes 2-5 minutes. This causes the 5-minute blackout. In BGP mode, all nodes advertising the Service IP withdraw the route immediately upon failure, and BGP convergence (with BFD) redirects traffic in under 1 second. The fix is to migrate from Layer 2 to BGP mode for production Services.

Question 10

What five things do you check, starting from the most likely cause?

Accepted Answer

1. Service name and namespace: verify with kubectl get svc. 2. CoreDNS is running: kubectl get pods -n kube-system -l k8s-app=kube-dns. 3. CoreDNS Corefile: Check if the kubernetes plugin is configured correctly. 4. Pod resolv.conf: should show nameserver 10.96.0.10 and search domains. 5. NetworkPolicy blocking UDP 53: A default-deny policy may block DNS queries.

Question 11

What architecture enables multi-cluster service discovery, and how does CoreDNS handle it?

Accepted Answer

The architecture is multi-cluster service mesh or federated DNS. Option 1: Istio multi-cluster with shared root CA. Istio creates ServiceEntries in Cluster A for Services in Cluster B, and the Istio control plane handles cross-cluster routing via east-west gateways. Option 2: CoreDNS with DNS forwarding. In Cluster A, I configure CoreDNS to forward queries for *.cluster-b.local to CoreDNS in Cluster B. Option 3: ExternalDNS + global load balancer. ExternalDNS syncs K8s Services to Route 53 or Cloudflare.

Question 12

What Go-specific networking behavior causes this, and what kernel parameter fixes it?

Accepted Answer

Go HTTP client uses HTTP keep-alive by default with a connection pool. If the server closes idle connections but the client reuses them, Go sends a request on a closed connection, resulting in TCP RST and retransmissions. This is the stale connection problem. The fix is to configure the Go HTTP transport with IdleConnTimeout shorter than the server idle timeout. From the infrastructure side, I also check conntrack exhaustion. If the node has many short-lived connections, the conntrack table fills up and new connections are dropped. I check with conntrack -L | wc -l and increase nf_conntrack_max if needed.

Question 13

Why does Flannel not support NetworkPolicy, and what is the migration path to a CNI that does?

Accepted Answer

Flannel is a simple overlay CNI that provides Pod networking but does not implement NetworkPolicy. It has no control plane for enforcing ingress/egress rules. NetworkPolicy requires a CNI with a policy engine: Calico, Cilium, Weave Net, or Antrea. The migration path from Flannel to Calico is: (1) Install Calico in policy-only mode initially: Calico enforces NetworkPolicy while Flannel continues to handle Pod networking. (2) Once validated, migrate Pod networking to Calico by changing the IP pool and restarting nodes one by one. (3) Remove Flannel. For zero-downtime migration, I use a canary node pool.

Question 14

What two Ingress features enable gRPC routing, and what is the alternative if the Ingress does not support HTTP/2?

Accepted Answer

gRPC requires HTTP/2 end-to-end because it uses HTTP/2 frames for streaming and multiplexing. Standard Kubernetes Ingress (NGINX) supports gRPC with two annotations: nginx.ingress.kubernetes.io/backend-protocol: "GRPC" and nginx.ingress.kubernetes.io/grpc-backend: "true". The Ingress controller must also have TLS termination because most gRPC clients require HTTPS. If the Ingress controller does not support HTTP/2, the alternative is a Layer 4 LoadBalancer (MetalLB, AWS NLB) that forwards TCP port 50051 directly to the Service without inspecting HTTP. Another alternative is Envoy Gateway or Istio Ingress Gateway, which natively support gRPC and advanced routing.

Question 15

What CNI do you recommend, and what three features make it suitable for this scale and workload diversity?

Accepted Answer

I recommend Cilium for this scale and diversity. Three features: (1) eBPF-based datapath — Cilium replaces iptables with eBPF, providing O(1) policy enforcement regardless of the number of NetworkPolicy rules. At 5,000 nodes with hundreds of policies per namespace, iptables-based CNIs suffer from O(n) rule evaluation and high CPU. (2) Native KubeVirt support — Cilium supports Layer 2 announcement and BGP advertisement for KubeVirt VMs, allowing VMs to have routable IPs via MetalLB or direct BGP peering with ToR switches. (3) Bandwidth management and observability — Cilium supports Pod-level bandwidth limiting and Hubble for flow visibility. For ML training workloads that require RDMA or high-throughput east-west traffic, Cilium supports Big TCP and eBPF-based load balancing that bypasses kube-proxy entirely.

Kubernetes Networking Interview Questions: CNI, Service Mesh & Troubleshooting

More Cloud Networking Interview Prep