Stand up for Chaos!

ChaosWheel

About the speaker

Aravind Putrevu

Aravind Putrevu

Developer Advocate,

Elastic

Aravind is passionate about evangelizing technology, meeting developers, and helping in solving their problems. He is a backend developer and has eight years of development experience. Currently, he works at Elastic as Developer Advocate and looks after the Developer Relation function of India, South East Asia. Previously, He worked at McAfee Antivirus as a Sr. Software Engineer in Cloud Security Domain.

About the talk

You might be running an eCommerce, Food Delivery, or a SaaS service. But the common possibility of all these risks or otherwise "Chaos" a simple outage might create to your customers, partners, or even internal platform users.

You might be running an eCommerce, Food Delivery, or a SaaS service. But the common possibility of all these risks or otherwise "Chaos" a simple outage might create to your customers, partners, or even internal platform users.

"Chaos" is an interesting concept of getting ready for the outage day and minimizing the blast radius by using specific methods.

In this talk, we'll take a look at the story of chaos in an application, "free & open" tools available, some best practices!

Transcript

Demonstration of Observability in Chaos

Aravind begins the demonstration by showing a slack notification titled "High Mean Transaction Duration for the Advert Service", stating that it is an automated response generated by a deployed application microservice. Then we proceed to understand the example with an e-commerce web application of which the "Advert Service" microservice is a part. Through the entirety of the website, users can browse through the different products that are being sold, read the description of any particular product, add one or more products to the cart, and finally place an order. The entire application is built using a microservice architecture as seen from an Elastic Kibana APM dashboard:

The architecture consists of a front-end layer connected to a Node.js application layer containing all the APIs such as the Java advert service, Golang checkout service, and the Python recommendation service, which are then again connected to a bunch of services. Aravind explains that every large enterprise seeks to perform a scale test before their targeted campaigns or events which draws many potential customers, and stresses the application infrastructure.

Aravind proceeds to demonstrate the details of the advertService, wherein one can observe the anomaly score of the service, the anomaly timeline of the service, and a bunch of different observability scores for the service that can be configured for different alerts. Further on, Aravind introduces the transactions for the advertService to diagnose the particular anomaly in the service, where one can see the methods being called by the service along with their timelines and other relevant details.

The APM services dashboard also lets us analyze specific services for the number of error rates for the duration of their execution, to better understand the error in the system, the performance impact created because of a specific load included in the system, and how the system is responding overall. It also lets us access the application-specific logs and compiler-specific metrics, say, JVM metrics such as CPU usage, system memory usage, heap memory usage, garbage collection rate, etc.

Another approach to resolving the system anomaly is using the Log, Metrics, and APM dashboard, where one can specifically check when the application has restarted. Similarly, we can observe multiple metrics for observing and figuring out any anomaly. One can also inspect the service deployment-specific errors to understand the nature of the anomaly. Based on this information, the application logs can be inspected for the requisite time period to better understand how the application execution took place. An alert can be put to this log stream using a set of desirable conditions.

Apart from the application viewpoint, one can also try to understand their system's anomalies from the infrastructure point of view, where one can inspect the infrastructure specific details of a system like the Kubernetes pods that are running or the Docker containers that are running, analyzing metrics such as memory usage or CPU usage. It's also possible to check the uptime for different services, checking their ping or availability or status, to determine their performance over different time duration or geo-location, and so on. This is all about incident analysis from the observability point of view, how to search for the changes in the system and compare them.

IBM's Principles of Chaos Engineering
IBM's Principles of Chaos Engineering
Ana Medina
In the kitchen: A sprinkle of fire and chaos

Sign Up

for our Newsletter

Get tips, best practices and updates on Chaos engineering in cloud native.

Videos

by Experts

Checkout our videos from the latest conferences and events

Our Videos

Related Blogs

Defining Chaos Engineering & Its Principles

Read

May 27, 2021

7 Min Read

Prithvi's Blog