TL;DR: Chaos Engineering means breaking things on purpose to learn how our systems behave when something goes wrong.
What is the worst-case scenario in your infrastructure? What would happen if it actually occurred? And more importantly, how prepared are you to handle it?
Chaos Engineering is the idea that we need to periodically break things in advance to see if our systems can handle the worst-case scenario. Have we overlooked anything? Are there any weaknesses we haven’t thought of?
High availability is just the beginning.
When building infrastructure, we usually think about redundancy. We add extra servers, configure load balancers, replicate data, and set up monitoring. The goal is to prevent a single point of failure from taking down the whole system.
For example, in Kubernetes, having three control-plane nodes helps a cluster maintain its control plane when one node goes down, assuming the cluster uses a three-member etcd configuration and the remaining components are healthy.
Sounds good, right?
And any ever actually tested what happens when one of those nodes goes down?
Does the cluster continue working as expected? Can workloads still run? Does the monitoring system alert us? And how long does it take before we notice something is wrong?
We can design our infrastructure to handle failures, but that (design) doesn’t prove that everything will work as expected.
To make sure that everything happens according is priority.
What makes Chaos Engineering?
In simple terms, instead of waiting for a production incident to reveal a weakness, we create controlled situations that help us discover problems before they become real disasters.
For example, we might:
- Stop an application and see whether it recovers automatically.
- Shut down a worker node and observe how Kubernetes handles the lost workloads.
- Make a service temporarily unavailable and check whether dependent applications can cope.
- Introduce network latency and see whether requests start timing out.
- Make a dependency unreachable and find out whether our application handles the situation gracefully.
The point isn’t to break things just for fun. The point is to learn something from the experience.
We want to know what happens, whether the system behaves as expected, and what we should fix if it doesn’t.
What Could Possibly Go Wrong?
Let’s take a simple example.
Imagine we have a web application running on Kubernetes. We have multiple replicas, a Service to distribute traffic, and a readiness probe to prevent unhealthy pods from receiving requests.
Everything looks good on paper.
Now, what happens if one of the pods suddenly disappears?
We expect Kubernetes to notice the missing pod and create a replacement. The remaining replicas should continue serving requests, assuming they have enough capacity and the application is healthy.
But what if the replacement takes longer than expected to start? What if the remaining replicas cannot handle the extra traffic? What if the application depends on another service that is already struggling?
One small failure can expose problems that weren’t obvious when everything was running normally.
This is where Chaos Engineering becomes useful. We can test these scenarios, observe the results, and fix the weaknesses we discover.
Start Small, Break Things Carefully
Of course, we shouldn’t randomly shut down servers in production and hope for the best.
Chaos Engineering needs a plan.
Before running an experiment, we should know what we want to test and what we expect to happen.
For example:
Experiment: Stop one application pod.
Expectation: Kubernetes replaces the pod, and the application remains available through its other healthy replicas.
What to observe:
- Does the application remain reachable?
- Do requests start failing?
- How long does recovery take?
- Do our monitoring and alerting systems report the event correctly?
After the experiment, we compare what actually happened with what we expected.
If everything works, that’s useful evidence that our assumptions hold under the conditions we tested. If something goes wrong, even better — we have found a weakness we can investigate before it causes an unexpected incident.
We should also have clear safety limits. Start in a development or staging environment, keep the experiment small, monitor the results, and have a way to stop the test if things get worse than expected.
Once we’re confident in the process, we can consider more realistic experiments in production with appropriate safeguards.
Then, this is not just Kubernetes
Although Kubernetes is a good place to start, Chaos Engineering isn’t limited to containers or orchestration platforms.
The same idea applies to almost any infrastructure.
Running Linux servers? Test what happens when a service stops unexpectedly.
Operating a database? Test how your application behaves when the database becomes temporarily unavailable.
Using a load balancer? Check whether traffic is redirected correctly when a backend becomes unhealthy.
Depending on an external API? Find out what happens when that API responds slowly or stops responding altogether.
Even a backup strategy deserves to be tested. Having backups is one thing; knowing that you can restore your data when you need it is another.
The scenarios will differ, but the question remains the same:
When something breaks, does our system behave the way we expect it to?
Learn From What Breaks
Chaos Engineering isn’t about proving that our infrastructure is perfect. No system is immune to every possible failure, and we can’t test every combination of problems.
Instead, it’s about reducing uncertainty.
Every experiment gives us an opportunity to learn more about our systems. We might discover that our recovery process takes too long, our alerts don’t work as expected, or our application cannot handle a dependency becoming unavailable.
Those findings give us something concrete to improve.
And after fixing the problem, we can run the experiment again to see whether the change actually helped.
Over time, this helps us build confidence in our infrastructure based on what we’ve observed, rather than what we assume.
What Is the Difference Between Chaos Engineering and Disaster Recovery Planning (DRP)?
Disaster Recovery Planning (DRP) focuses on how we recover when a major disaster happens, such as losing a data center, suffering a major outage, or losing access to critical data.
Chaos Engineering, on the other hand, focuses on breaking things on purpose to see how our systems behave when something goes wrong. It can be something small, like stopping an application, or something bigger, like losing a server or a network connection.
In other words, DRP helps us prepare for recovery from major disasters, while Chaos Engineering helps us discover weaknesses before they turn into bigger problems.
We need to prepare for both situations. Having a recovery plan is important, but we also need to test our assumptions and make sure our systems behave as expected when things break.
Back to Certified Cloud Native Platform Engineering Associate Preparation