How Kubernetes Actually Works: Pods, Controllers, and the Reconciliation Loop

Kubernetes is the dominant platform for container orchestration, yet many of its users interact with it through a fog of abstraction.^[24] They know kubectl apply makes things happen, but the underlying mechanism remains a mystery. This isn't just an academic detail. Understanding how Kubernetes actually works, from its core philosophy down to the specific interactions between its components, is essential for designing resilient systems, debugging failures, and making sound architectural decisions.^[25] At its heart, Kubernetes is not a magical black box; it's a distributed control system built on a simple and powerful pattern: continuously driving the current state of a system toward a user-defined desired state.

This system's design philosophy is a direct descendant of Google's internal cluster manager, Borg^[1]. When Google open-sourced Kubernetes in mid-2014^[2], it brought Borg's declarative, automation-centric approach to the world. Instead of issuing imperative commands like "run this container," you declare the state you want: "I need three replicas of this application running." The rest is the work of a tireless automated process called the reconciliation loop, orchestrated by a set of components known as controllers. This article explains precisely how that process works.

The Core Idea: Desired State Management

To understand Kubernetes, you must first grasp its declarative nature. The entire system revolves around a central, reliable data store called etcd^[3]. This key-value store is the single source of truth for the cluster. Every object in Kubernetes, from a Pod to a Deployment to a Secret, is represented as a record in etcd. Critically, each object has two main parts: a spec and a status.

• spec (Specification): This is the desired state. It's the part you, the user, define. It describes what you want the object to look like and how you want it to behave. For example, a Deployment's spec would include the container image to use and the number of replicas you want running.
• status: This is the observed state. It's written and updated by the Kubernetes system itself, not by the user. It describes what is actually happening in the cluster at this moment. For a Deployment, the status would show how many replicas are currently running and available.

This separation is the foundation of Kubernetes's control model. You declare your intent in the spec field^[4] by writing it to the Kubernetes API Server^[5], which then saves it to etcd. Specialized processes called controllers then observe the cluster, compare the spec (desired state) with the status (observed state), and take action to make the observed state match the desired state. This endless cycle of checking and correcting is the reconciliation loop.

The Atomic Unit: What is a Pod?

Before we can understand how controllers work, we must understand what they manage. In Kubernetes, the smallest and most fundamental deployable unit is not a container; it's a Pod^[6]. A Pod is a group of one or more containers that are scheduled together on the same host and share certain resources. Specifically, all containers within a Pod share:

• A network namespace: They all have the same IP address and can communicate with each other over localhost.
• Storage volumes: They can share data through a common set of attached storage volumes.
• Process namespace (optionally): They can see and signal each other's processes.

This design directly addresses a common pattern: tightly coupled helper processes. Imagine a main application container that needs a separate container to handle logging, another for a service mesh proxy, and another to export metrics. Instead of violating the "one process per container^[7]" best practice by stuffing all these functions into a single container image, you group them into a single Pod. They are managed as a single atomic unit, sharing a lifecycle and a local network, which is a much cleaner abstraction than managing individual containers with complex linking and networking. The use of Linux namespaces is what makes this lightweight grouping possible without the overhead of a full virtual machine.

A Pod is defined in a YAML manifest. Here’s an example of a Pod that runs an Nginx web server alongside a Fluentd container for log shipping.

apiVersion: v1
kind: Pod
metadata:
  name: web-and-logger
spec:
  containers:
  - name: web-server
    image: nginx:1.25
    ports:
    - containerPort: 80
  - name: log-shipper
    image: fluent/fluentd:v1.16

Crucially, Pods are considered mortal. They have a defined lifecycle^[8]: they are created, assigned to a node, they run, and then they exit. When a Pod dies, it is not brought back to life. This ephemerality is a feature, not a bug. To achieve reliability and scaling, we don't care about the fate of any single Pod. Instead, we use a higher-level object, a controller, to manage a set of identical, replaceable Pods.

The Engine of Automation: Controllers and the Reconciliation Loop

The magic of Kubernetes self-healing and scaling lies in its controllers^[9]. A controller is a software loop that tracks at least one Kubernetes resource type. These loops are bundled into a single binary and process called the kube-controller-manager^[10], a core component of the Kubernetes control plane. Each controller is responsible for a specific domain. The Node controller manages node status. kube-proxy is a standalone network proxy that runs on each node, not a controller bundled in kube-controller-manager, and the Service controller for cloud load balancers is part of the cloud-controller-manager. The most common controllers for application workloads are Deployments and ReplicaSets.

Let's trace the exact sequence of events when you deploy an application. This step-by-step process is the reconciliation loop in action.

Step 1: The User Defines the Desired State
You create a YAML file for a Deployment, specifying you want three replicas of your application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template: # This is the Pod spec for the Pods this Deployment will create
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: my-app-image:1.0.0

You apply this manifest using kubectl apply -f my-app.yaml. kubectl is simply an HTTP client that sends this YAML to the Kubernetes API Server.

Step 2: The API Server Writes to Etcd
The API Server receives the request. It authenticates you, validates the YAML to ensure it's well-formed, and then writes this Deployment object into etcd. At this moment, the desired state is officially recorded. The spec says replicas: 3, but the status shows 0 replicas are running.

Step 3: The Deployment Controller Wakes Up
The Deployment controller is constantly watching the API Server for changes to Deployment objects. It uses a mechanism called a "watch^[11]" for this. It immediately gets a notification that the my-app Deployment has been created or updated. The controller's reconciliation loop for my-app is now active.

Step 4: The Deployment Controller Reconciles
The controller reads the my-app Deployment. It sees spec.replicas is 3. It then checks for a ReplicaSet^[12] that it manages. A Deployment^[13] doesn't create Pods directly; it outsources that job to a ReplicaSet to facilitate rolling updates. Finding no existing ReplicaSet for this version of the application, the Deployment controller's job is to reconcile this difference. It creates a new ReplicaSet object, setting its spec.replicas to 3 and copying the Pod template from the Deployment. It then sends this new ReplicaSet object to the API Server, which saves it to etcd.

Step 5: The ReplicaSet Controller Wakes Up
Now, a different controller takes over. The ReplicaSet controller, which is also watching the API Server, gets a notification about the new ReplicaSet. Its reconciliation loop for this new object becomes active.

Step 6: The ReplicaSet Controller Reconciles
The ReplicaSet controller reads its new object and sees spec.replicas is 3. It then queries the API Server for all Pods matching its label selector (app: my-app). It finds 0. The desired state (3 Pods) does not match the current state (0 Pods). Its job is to create three Pods to fix this. It uses the Pod template within its spec to generate three Pod objects and sends them to the API Server, which saves them to etcd.

Step 7: The Scheduler Assigns Nodes
A third control plane component, the kube-scheduler^[14], sees three new Pods that have been created but have no nodeName field. Its sole job is to assign new Pods to a worker node. It evaluates all available nodes based on resource requests, affinity rules, and other constraints, and picks the best node for each Pod. It does not run the Pod. The scheduler does not update etcd directly; it sends a Binding object to the API Server, which then updates the Pod in etcd.

Step 8: The Kubelet Creates the Containers
On every worker node in the cluster runs an agent called the kubelet^[15]. Each kubelet is also watching the API Server. It's programmed to only pay attention to Pods that have been assigned to its node. The kubelet on the assigned node sees a new Pod object with its nodeName appear. It reads the Pod's spec, which includes the container definitions.

It then instructs the Container Runtime^[16] on the node (like containerd^[17]) to:

1. Pull the required container images (my-app-image:1.0.0).
2. Create the containers.
3. Set up the shared Pod network namespace.
4. Mount any necessary volumes.
5. Start the processes inside the containers.

Step 9: The Loop Closes
Once the containers are running, the kubelet continuously reports their status back to the API Server. The API Server updates the status field of the Pod objects in etcd. The ReplicaSet controller sees that 3 Pods are now running, and the Deployment controller sees that its ReplicaSet is at full strength. The observed state now matches the desired state, and the reconciliation loops become quiescent, simply waiting for the next change.

A Concrete Example: Surviving a Node Failure

The power of this model becomes clear during a failure. Imagine one of the three worker nodes hosting a Pod for my-app suddenly powers off.

1. Detection: The kubelet on that node stops sending its regular heartbeat to the control plane. After a timeout, the Node controller marks the Node's status as NotReady^[18].
2. Eviction: When a node becomes NotReady, the node controller taints it with a NoExecute effect. Pods are then evicted if they do not tolerate this taint. A default tolerationSeconds of 300 seconds (5 minutes) is automatically added to most pods for the not-ready and unreachable taints, which results in a five-minute grace period before the Pod is marked for deletion (evicted), though it remains in etcd in a Terminating state until the kubelet confirms termination or it is force-deleted.
3. Reconciliation: The ReplicaSet controller is still watching. It suddenly gets a notification that one of its Pods has been deleted. It re-counts the Pods matching its selector and now finds only 2. Its desired state (replicas: 3) no longer matches its observed state (2). The reconciliation loop kicks in.
4. Self-Healing: The ReplicaSet controller creates a new Pod object to replace the missing one and sends it to the API Server.
5. Rescheduling: The scheduler sees this new, unscheduled Pod and assigns it to a healthy, available worker node.
6. Recovery: The kubelet on that healthy node sees the new Pod assigned to it and starts the container. The application is back to 3 replicas without any human intervention.

Common Misconceptions

• Misconception 1: "Kubernetes restarts my failed container."
  This is only partially true. If a process inside a container crashes, the kubelet on the node will restart it according to the Pod's restartPolicy^[19]. But if the entire Pod fails (e.g., the node dies, or the Pod is evicted), it's the controller (like a ReplicaSet) that replaces the ephemeral Pod with a brand new one. The old Pod is gone forever.

• Misconception 2: "Pods are like small VMs."
  No. Pods do not have their own kernel. They are a logical grouping of containers that share Linux namespaces. This makes them significantly more lightweight in terms of resource usage and startup time compared to virtual machines.

• Misconception 3: "kubectl apply runs my application."
  kubectl does not run anything. kubectl sends an HTTP request to the API Server, which is the only component that writes directly to the etcd database. The actual work is performed asynchronously by the cluster's controllers and kubelets, which are constantly watching for changes in that database. This is why the command returns almost instantly, even for a complex deployment that may take minutes to fully roll out.

The Real Tradeoffs and Decision Lens

Kubernetes is powerful, but it is not a universal solution. Adopting it involves significant tradeoffs that every technical leader must weigh.^[26]

Costs and Complexity: The primary cost is complexity. Kubernetes introduces many layers of abstraction (Pods, Services, Ingresses, Deployments) which, while powerful, create a steep learning curve and can make debugging difficult. There is also resource overhead; the control plane components and node agents consume CPU and memory that could otherwise be used by applications. Managing a production-grade Kubernetes cluster requires significant operational expertise, covering areas like networking, storage, security, and monitoring.^[27][24]

When to Use Kubernetes:

• Microservices and High Availability: If you are managing a fleet of microservices and require automatic self-healing, scaling, and service discovery, Kubernetes is purpose-built for the job.^[28]
• Cloud Portability: It provides a consistent API layer across different cloud providers (Amazon EKS^[20], Google Kubernetes Engine^[21], Azure Kubernetes Service^[22]) and on-premises infrastructure, preventing vendor lock-in for your application deployments.^[28]
• Standardized CI/CD: When you need a standardized platform for development, testing, and production environments that enables repeatable, automated deployments.^[29]
• Sufficient Team or Managed Services: Your organization has a dedicated platform or SRE team, or you are committing to using a managed Kubernetes offering from a cloud provider, which offloads the control plane management burden.

When to Avoid Kubernetes:

• Simple Applications: For a simple monolithic application or a handful of services, the operational overhead of Kubernetes can far outweigh its benefits. A few VMs or a simpler PaaS (Platform as a Service) might be a better fit.
• Lack of Expertise: If you don't have the in-house expertise to run it and are not using a managed service, a poorly configured or unmaintained Kubernetes cluster can become a liability, not an asset.
• Unsuitable Workloads: While Kubernetes can run stateful applications like databases using StatefulSets^[23] and persistent storage, it adds significant complexity. Legacy applications not designed for ephemeral, containerized environments may be difficult to adapt and better suited for traditional infrastructure.

Where Kubernetes Stands Today

The core APIs for workloads like Deployments and Pods are stable and mature. The frontier of development has moved to the surrounding ecosystem, with intense innovation in areas like service meshes (Istio, Linkerd), GitOps tools (ArgoCD, Flux), and security policies. For a technical leader today, the question is rarely "Should we use containers?" but rather "How should we adopt Kubernetes?" Understanding its fundamental control loop is the first and most important step in answering that question correctly.