Chaos Engineering: Enhancing Business Service Reliability Through Failure...

Effective IT system management is crucial for the success of many companies, particularly large ones. Every minute of downtime in a high-load system can result in customer loss and damage to a brand’s reputation. According to Gremlin, businesses lose millions of dollars per hour during system outages.

Fewer than one half of data center owners and operators are tracking the metrics needed to assess their sustainability and/or meet pending regulatory requirements.

The frequency and severity of data center outages remain mainly unchanged from 2023 or show small improvements. Operators are countering increases in complexity, density and extreme weather with investment and good management practices.

Enterprises continue to meet their IT needs with hybrid architectures. More than one half of workloads (55%) are now off-premises, continuing the gradual trend of recent years.

— Uptime’s 14th Annual Global Data Center Survey

One method to reduce losses from technical incidents is chaos engineering, a practice widely adopted by major tech companies worldwide. So, how does it work, and what is the impact?

This article will be valuable for entrepreneurs looking to stay informed about proven tools for safeguarding IT infrastructure and applications.

Challenges Addressed by Chaos Engineering
Chaos Engineering Failure Simulation
Common Causes of Service Outages and When Chaos Engineering Helps
Who Conducts Failure Simulations and What Skills Are Needed
Advantages and Limitations of Chaos Engineering
- Advantages
- Limitations
Calculating the Economic Impact of Chaos Engineering
Summary

Challenges Addressed by Chaos Engineering

Chaos Engineering is the practice of intentionally creating failure scenarios in business services to enhance their reliability, helping to avoid reputational and financial losses.

This practice enables you to:

Test application resilience.
Identify weak points and hidden issues in design and scaling.
Improve system performance under real-world conditions.

This is particularly important for online services such as financial and healthcare institutions, telecommunications, transportation companies, e-commerce, and social networks.

Testing includes simulating failures of server components, network infrastructure, or specific applications. There exists a standard testing model developed by global IT companies and the international Awesome Chaos Engineering community.

Chaos Engineering Failure Simulation

Define the stable state of the business service

This step sets a baseline to return to after the failure simulation. Any deviation from the norm is measured against this stable state.

Formulate a hypothesis based on real events

These can include server crashes, faulty hard drives, or network disruptions. Past incidents and known vulnerabilities in the tested technology serve as a basis for hypothesis creation.

Simulate the failure

Document all events that occur during the experiment. This data will guide decisions about potential application failures.

Automate the experiment and repeat it continuously

Each new version of a business service may reveal new hidden failures. Continuous simulation helps protect new versions from potential issues.

Common Causes of Service Outages and When Chaos Engineering Helps

According to a 2023 Uptime Institute survey of 600 companies across various sectors, over one-third experienced major outages in the last three years, with most experiencing minor outages.

Chaos Engineering: Enhancing Business Service Reliability Through Failure Simulation

Chaos engineering helps address the main causes of outages:

Network issues
Power failures leading to server reboots
IT system or software malfunctions
Failures in external IT providers' services

As network outages and power related issues account for more than a half of all incidents, it becomes easy to reduce the risk of losses by choosing datacenters with a record of 14 years of five nines (99.999%) uptime.

Who Conducts Failure Simulations and What Skills Are Needed

A chaos engineer is more of a role than a job title. DevOps engineers, developers, support engineers, or system administrators can conduct experiments.

Required technical skills include:

Proficiency in administering Linux or Windows operating systems
Experience with cloud services, often based on Kubernetes or OpenShift platforms
Programming skills for writing scripts to simulate failures, such as bash scripts in Linux

Advantages and Limitations of Chaos Engineering

Advantages	Limitations
Controlled experiments identify vulnerabilities in business services, increasing their resilience. Chaos Engineering helps uncover system performance bottlenecks, enhancing service functionality. It provides the development team with a deeper understanding of how their system performs under stress, improving monitoring and diagnostics. Teams gain better insight into the service’s weaknesses and ways to improve it, fostering professional growth. Chaos Engineering can be applied to services with various architectures—whether they are new cloud-based services or those designed over 10 years ago.	Requires restructuring testing processes and making changes to the system architecture. Demand significant effort and time from the development team, introducing new protocols, developing and training personnel, and reworking the business service architecture. Requires additional human resources Can lead to temporary or extended disruptions in system performance, lasting from 10 minutes to 3–5 days. While this is the practice's purpose, users may experience temporary issues. In rare cases, failure simulations can cause service unavailability, especially with databases, which might take several days to recover. It's crucial to thoroughly analyze which system components were affected and what issues were identified, requiring in-depth knowledge of the system's architecture and its components.

Advantages

Limitations

Controlled experiments identify vulnerabilities in business services, increasing their resilience.
Chaos Engineering helps uncover system performance bottlenecks, enhancing service functionality.
It provides the development team with a deeper understanding of how their system performs under stress, improving monitoring and diagnostics.
Teams gain better insight into the service’s weaknesses and ways to improve it, fostering professional growth.
Chaos Engineering can be applied to services with various architectures—whether they are new cloud-based services or those designed over 10 years ago.

Requires restructuring testing processes and making changes to the system architecture.
Demand significant effort and time from the development team, introducing new protocols, developing and training personnel, and reworking the business service architecture.
Requires additional human resources
Can lead to temporary or extended disruptions in system performance, lasting from 10 minutes to 3–5 days. While this is the practice's purpose, users may experience temporary issues.
In rare cases, failure simulations can cause service unavailability, especially with databases, which might take several days to recover. It's crucial to thoroughly analyze which system components were affected and what issues were identified, requiring in-depth knowledge of the system's architecture and its components.

Calculating the Economic Impact of Chaos Engineering

First, consider the cost of service downtime. The figure depends on the scale, complexity, and specifics of the IT system. On average, one minute of downtime in a high-load system costs between $7,200 and $9,000, according to Uptime Institute and Gremlin.

To calculate the economic impact of a technical failure and the costs of using chaos engineering, consider the following scenario: a business launches a new product, invests in advertising, and traffic increases, leading to equipment overload and service failure.

Costs of incident resolution

Losses from the incident: Foregone profit
Emergency response team: Payment for 5-10 employees
Temporary solution development: Daily wages for the development team
Permanent solution development: Up to 14 days of developer wages

In contrast, chaos engineering would involve

Simulation cost: Payment for 1-2 employees
Permanent solution development: Up to 14 days of developer wages

In this case, investing in chaos engineering would cost 2-3 times less than the cost of an actual failure.

Summary

Chaos engineering is an IT practice that intentionally creates failure scenarios in business services to improve their reliability. This practice helps identify hidden problems in design, scaling, and fault tolerance, ultimately reducing financial losses and risks during system failures.

The practice is relevant whether you are choosing between on-premises server location and cloud infrastructure, using multi-cloud strategy.

Chaos engineering is especially relevant for online services like financial institutions, healthcare, telecommunications, transportation, e-commerce, and social networks.

The cost of conducting failure simulations can range widely depending on the business system's size and complexity, while the cost of actual failures can reach tens millions of dollars.