TL;DR: Reconciliation is Kubernetes continuously checking the actual state against the desired state and taking action when they are different.

Imagine that you need to keep a certain number of copies of the same thing.

Sometimes you need to create 100 copies. Sometimes you need to delete them and leave only one copy. If you do this manually, you need to keep checking the status, count how many copies currently exist, and then create or delete copies to reach the number you need.

Automation can handle this process for you.

In Kubernetes, this process has a special term: “reconciliation”, or the “reconciliation loop”.

Kubernetes continuously checks the current state against the desired state and takes action when they do not match.

Simple Kubernetes (Stateless)

We will begin with a simple one, Stateless Kubernetes.

In a stateless application, Pods are usually interchangeable. If we need more Pods, Kubernetes creates them. If we have too many, Kubernetes removes some. If a Pod fails, Kubernetes works to replace it.

If the desired number of replicas is 3, Kubernetes needs to maintain three running Pods.

If there are four Pods instead then remove 1 Pod.

The controller continuously observes the state of the cluster through the Kubernetes API and compares the current state with the desired state.

If the number of Pods is lower than the desired number, it creates more. If the number is higher, it removes some. If a Pod fails, the controller takes the appropriate action to bring the application back toward the desired state.

This continuous process of observing, comparing, and acting is called reconciliation.

More Headache Kubernetes (Stateful)

Now we discuss more advanced Kubernetes with Custom APIs (CRDs) and applications that have state, such as StatefulSets and Operators.

This kind of Kubernetes may need more care because Pods are not always interchangeable, and the controller may need to understand application-specific state, storage, identity, or dependencies.

It can be more headache, but the basic concept is still the same:

{Observe > Compare > Act > Observe > ...} = Reconcile

The difference is that there are more things to observe, more things to compare, and potentially more complicated actions to take.

What Kind of Add-ons Do We Need for This More Headache One?

When Kubernetes becomes more complicated, we may need some additional pieces to help Kubernetes understand what we want and how to manage it.

For example:

  • StatefulSet — when Pods need stable identity and persistent storage.

  • Custom Resource Definition (CRD) — when the standard Kubernetes API is not enough and we need to define our own resource.

  • Controller — watches a resource and tries to make the actual state match the desired state.

  • Operator — a controller with application-specific knowledge, usually working with a CRD.

  • Policy Engine — adds rules about what is allowed or how resources should behave.

  • Crossplane — extends the Kubernetes control-plane model to manage external infrastructure and cloud resources.

These are different tools and concepts, but they have something in common: they add more knowledge or control to the Kubernetes reconciliation process.

Instead of only asking:

“Do I have three Pods?”

we may need to ask:

“Is my application configured correctly? Is the storage attached? Is the database ready? Are the dependencies available? Does this resource follow the required policy?”

That is where Kubernetes starts becoming a little more headache.

I think this YouTube video gives a good concept of how this works:

Back to Certified Cloud Native Platform Engineering Associate Preparation