Bottleneck #05: Resilience and Observability

Use multiple zones within a region

For highly critical services (and their data), configure and enable
them to run across multiple zones. This should give a bump to your
system availability, and increase your resiliency in the case of
disruption (within a zone).

Specify appropriate computing instance types and specifications

Business critical services should have computing capacity
appropriately assigned to them. If services are required to run 24/7,
your infrastructure should reflect those requirements.

Match investment to critical service tiers

Many organizations manage investment by identifying critical
service tiers, with the understanding that not all business systems
share the same importance in terms of delivering customer experience
and supporting revenue. Identifying service tiers and associated
resilience outcomes informed by service level agreements (SLAs), paired with architecture and
design patterns that support the outcomes, provides helpful guardrails
and governance for your product development teams.

Clearly define owners across your entire system

Each service that exists within your system should have
well-defined owners. This information can be used to help direct issues
to the right place, and to people who can effectively resolve them.
Implementing a developer portal which provides a software services
catalog with clearly defined team ownership helps with internal
communication patterns.

Automate manual resilience processes (within a timebox)

Certain resilience problems that have been solved by hand can be
automated: actions like restarting a service, adding new instances or
restoring database backups. Many actions are easily automated or simply
require a configuration change within your cloud service provider.
While in the bottleneck, implementing these capabilities can give the
team the relief it needs, providing much needed breathing room and
time to solve the root cause(s).

Make sure to keep these implementations at their simplest and
timeboxed (couple of days at max). Bear in mind these started out as
bandaids, and automating them is just another (albeit better) type of
bandaid. Integrate these into your monitoring solution, allowing you
to remain aware of how frequently your system is automatically recovering and how long it
takes. At the same time, these metrics allow you to prioritize
moving away from reliance on these bandaid solutions and make your
whole system more robust.

Centralize your logs to be viewable through a single interface

Aggregate logs from core services and systems to be available
through a central interface. This will keep them accessible to
multiple eyes easily and reduce troubleshooting efforts (potentially
improving mean time to recovery).

Define a clear structured format for log messages

Anyone who’s had to parse through aggregated log messages can tell
you that when multiple services follow differing log structures it’s
an incredible mess to find anything. Every service just ends up
speaking its own language, and only the original authors understand
the logs. Ideally, once those logs are aggregated, anyone from
developers to support teams should be able to understand the logs, no
matter their origin.

Structure the log messages using an organization-wide standardized
format. Most logging tools support a JSON format as a standard, which
enables the log message structure to contain metadata like timestamp,
severity, service and/or correlation-id. And with log management
services (through an observability platform), one can filter and search across these
properties to help debug bottleneck issues. To help make search more
efficient, prefer fewer log messages with more fields containing
pertinent information over many messages with a small number of
fields. The actual messages themselves may still be unique to a
specific service, but the attributes associated with the log message
are helpful to everyone.

Treat your log messages as a key piece of information that is
visible to more than just the developers that wrote them. Your support team can
become more effective when debugging initial customer queries, because
they can understand the structure they are viewing. If every service
can speak the same language, the barrier to provide support and
debugging assistance is removed.

Add observability that’s close to your customer experience

What gets measured gets managed.

— Peter Drucker

Though infrastructure metrics and service message logs are
useful, they are fairly low level and don’t provide any context of
the actual customer experience. On the other hand, customer
notifications are a direct indication of an issue, but they are
usually anecdotal and don’t provide much in terms of pattern (unless
you put in the work to find one).

Monitoring core business metrics enables teams to observe a
customer’s experience. Typically defined through the product’s
requirements and features, they provide high level context around
many customer experiences. These are metrics like completed
transactions, start and stop rate of a video, API usage or response
time metrics. Implicit metrics are also useful in measuring a
customer’s experiences, like frontend load time or search response
time. It’s crucial to match what is being observed directly
to how a customer is experiencing your product. Also
important to note, metrics aligned to the customer experience become
even more important in a B2B environment, where you might not have
the volume of data points necessary to be aware of customer issues
when only measuring individual components of a system.

At one client, services started to publish domain events that
were related to the product experience: events like added to cart,
failed to add to cart, transaction completed, payment approved, etc.
These events could then be picked up by an observability platform (like
Splunk, ELK or Datadog) and displayed on a dashboard, categorized and
analyzed even further. Errors could be captured and categorized, allowing
better problem solving on errors related to unexpected customer
experience.

Figure 2:
Example of what a dashboard focusing on the user experience could look like

Data gathered through core business metrics can help you understand
not only what might be failing, but where your system thresholds are and
how it manages when it’s outside of that. This gives further insight into
how you might get through the bottleneck.

Provide product status insight to customers using status indicators

It can be difficult to manage incoming customer inquiries of
different issues they are facing, with support services quickly finding
they are fighting fire after fire. Managing issue volume can be crucial
to a startup’s success, but within the bottleneck, you need to look for
systemic ways of reducing that traffic. The ability to divert call
traffic away from support will give some breathing room and a better chance to
solve the right problem.

Service status indicators can provide customers the information they are
seeking without having to reach out to support. This could come in
the form of public dashboards, email messages, or even tweets. These can
leverage backend service health and readiness checks, or a combination
of metrics to determine service availability, degradation, and outages.
During times of incidents, status indicators can provide a way of updating
many customers at once about your product’s status.

Building trust with your customers is just as important as creating a
reliable and resilient service. Providing methods for customers to understand
the services’ status and expected resolution timeframe helps build
confidence through transparency, while also giving the support staff
the space to problem-solve.

Understand the costs of service failure

For a startup, the consequences of not hitting a revenue target
this ‘quarter’ might be different than for a scaleup or a mature
product. But as often happens, the initial “new features are more
valuable than technical debt” decision becomes a permanent fixture in the
organizational culture – whether the actual revenue impact is provable
or not; or even calculated. An aspect of the maturity needed when
moving from startup to scaleup is in the data-driven element of
decision-making. Is the organization tracking the value of every new
feature shipped? And is the organization analyzing the operational
investments as contributing to new revenue rather than just a
cost-center? And are the costs of an outage or recurring outages known
both in terms of wasted internal labor hours as well as lost revenue?
As a startup, in most of these regards, you’ve got nothing to lose.
But this is not true as you grow.

Therefore, it’s important to start analyzing the costs of service
failures as part of your overall product management and revenue
recognition value stream. Understanding your revenue “velocity” will
provide an easy way to quantify the direct cost-per-minute of
downtime. Tracking the costs to the team for everyone involved in an
outage incident, from customer support calls to developers to management
to public relations/marketing and even to sales, can be an eye-opening experience.
Add on the opportunity costs of dealing with an outage rather than
expanding customer outreach or delivering new features and the true
scope and impact of failures in resilience become apparent.

Manage resilience as a feature

Start treating resilience as more than just a technical
expectation. It’s a core feature that customers will come to expect.
And because they expect it, it should become a first class
consideration among other features. Part of this evolution is about shifting where the
responsibility lies. Instead of it being purely a responsibility for
tech, it’s one for product and the business. Multiple layers within
the organization will need to consider resilience a priority. This
demonstrates that resilience gets the same amount of attention that
any other feature would get.

Close collaboration between
the product and technology is vital to make sure you’re able to
set the correct expectations across story definition, implementation
and communication to other parts of the organization. Resilience,
though a core feature, is still invisible to the customer (unlike new
features like additions to a UI or API). These two groups need to
collaborate to ensure resilience is prioritized appropriately and
implemented effectively.

The objective here is shifting resilience from being a reactionary
concern to a proactive one. And if your teams are able to be
proactive, you can also react more appropriately when something
significant is happening to your business.

Requirements should reflect realistic expectations

Knowing realistic expectations for resilience relative to
requirements and customer expectations is key to keeping your
engineering efforts cost effective. Different levels of resilience, as
measured by uptime and availability, have vastly different costs. The
cost difference between “three nines” and “four nines” of availability
(99.9% vs 99.99%) may be a factor of 10x.

It’s important to understand your customer requirements for each
business capability. Do you and your customers expect a 24x7x365
experience? Where are your customers
based? Are they local to a specific region or are they global?
Are they primarily consuming your service via mobile devices, or are
your customers integrated via your public API? For example, it is an
ineffective use of capital to provide 99.999% uptime on a service delivered via
mobile devices which only enjoy 99.9% uptime due to cell phone
reliability limits.

These are important questions to ask
when thinking about resilience, because you don’t want to pay for the
implementation of a level of resiliency that has no perceived customer
value. They also help to set and manage
expectations for the product being built, the team building and
maintaining it, the folks in your organization selling it and the
customers using it.

Feel out your problems first and avoid overengineering

If you’re solving resiliency problems by hand, your first instinct
might be to just automate it. Why not, right? Though it can help, it’s most
effective when the implementation is time-boxed to a very short period
(a couple of days at max). Spending more time will likely lead to
overengineering in an area that was actually just a symptom.
A large amount of time, energy and money will be invested into something that is
just another bandaid and most likely is not sustainable, or even worse,
causes its own set of second-order challenges.

Instead of going straight to a tactical solution, this is an
opportunity to really feel out your problem: Where do the fault lines
exist, what is your observability trying to tell you, and what design
choices correlate to these failures. You may be able to discover those
fault lines through stress, chaos or exploratory testing. Use this
opportunity to your advantage to discover other system stress points
and determine where you can get the largest value for your investment.

As your business grows and scales, it’s critical to re-evaluate
past decisions. What made sense during the startup phase may not get
you through the hypergrowth stages.

Leverage multiple techniques when gathering requirements

Gathering requirements for technically oriented features
can be difficult. Product managers or business analysts who are not
versed in the nomenclature of resilience can find it hard to
understand. This often translates into vague requirements like “Make x service
more resilient” or “100% uptime is our goal”. The requirements you define are as
important as the resulting implementations. There are many techniques
that can help us gather those requirements.

Try running a pre-mortem before writing requirements. In this
lightweight activity, individuals in different roles give their
perspectives about what they think could fail, or what is failing. A
pre-mortem provides valuable insights into how folks perceive
potential causes of failure, and the related costs. The ensuing
discussion helps prioritize things that need to be made resilient,
before any failure occurs. At a minimum, you can create new test
scenarios to further validate system resilience.

Another option is to write requirements alongside tech leads and
architecture SMEs. The responsibility to create an effective resilient system
is now shared amongst leaders on the team, and each can speak to
different aspects of the design.

These two techniques show that requirements gathering for
resilience features isn’t a single responsibility. It should be shared
across different roles within a team. Throughout every technique you
try, keep in mind who should be involved and the perspectives they bring.

Broadly look at your architecture and determine appropriate trade-offs

Either implicitly or explicitly, when the initial architecture was
created, trade-offs were made. During the experimentation and gaining
traction phases of a startup, there is a high degree of focus on
getting something to market quickly, keeping development costs low,
and being able to easily modify or pivot product direction. The
trade-off is sacrificing the benefits of resilience
that would come from your ideal architecture.

Take an API backed by Functions as a Service (FaaS). This approach is a great way to
create something with little to no management of the infrastructure it
runs on, potentially ticking all three boxes of our focus area. On the
other hand, it’s limited based on the infrastructure it’s allowed to
run on, timing constraints of the service and the potential
communication complexity between many different functions. Though not
unachievable, the constraints of the architecture may make it
difficult or complex to achieve the resilience your product needs.

As the product and organization grows and matures, its constraints
also evolve. It’s important to acknowledge that early design decisions
may no longer be appropriate to the current operating environment, and
consequently new architectures and technologies need to be introduced.
If not addressed, the trade-offs made early on will only amplify the
bottleneck within the hypergrowth phase.

Enhance resilience with effective error recovery strategies

Data gathered from monitors can show where high failure
rates are coming from, be it third-party integrations, backed-up queues,
backoffs or others. This data can drive decisions on what are
appropriate recovery strategies to implement.

Use caching where appropriate

When retrieving information, caching strategies can help in two
ways. Primarily, they can be used to reduce the load on the service by
providing cached results for the same queries. Caching can also be
used as the fallback response when a backend service fails to return
successfully.

The trade-off is potentially serving stale data to customers, so
ensure that your use case is not sensitive to stale data. For example,
you wouldn’t want to use cached results for real-time stock price
queries.

Use default responses where appropriate

As an alternative to caching, which provides the last known
response for a query, it is possible to provide a static default value
when the backend service fails to return successfully. For example,
providing retail pricing as the fallback response for a pricing
discount service will do no harm if it is better to risk losing a sale
rather than risk losing money on a transaction.

Use retry strategies for mutation requests

Where a client is calling a service to effect a change in the data,
the use case may require a successful request before proceeding. In
this case, retrying the call may be appropriate in order to minimize
how often error management processes need to be employed.

There are some important trade-offs to consider. Retries without
delays risk causing a storm of requests which bring the whole system
down under the load. Using an exponential backoff delay mitigates the
risk of traffic load, but instead ties up connection sockets waiting
for a long-running request, which causes a different set of
failures.

Use idempotency to simplify error recovery

Clients implementing any type of retry strategy will potentially
generate multiple identical requests. Ensure the service can handle
multiple identical mutation requests, and can also handle resuming a
multi-step workflow from the point of failure.

Design business appropriate failure modes

In a system, failure is a given and your goal is to protect the end
user experience as much as possible. Specifically in cases that are
supported by downstream services, you may be able to anticipate
failures (through observability) and provide an alternative flow. Your
underlying services that leverage these integrations can be designed
with business appropriate failure modes.

Consider an ecommerce system supported by a microservice
architecture. Should downstream services supporting the ordering
function become overwhelmed, it would be more appropriate to
temporarily disable the order button and present a limited error
message to a customer. While this provides clear feedback to the user,
Product Managers concerned with sales conversions might instead allow
for orders to be captured and alert the customer to a delay in order
confirmation.

Failure modes should be embedded into upstream systems, so as to ensure
business continuity and customer satisfaction. Depending on your
architecture, this might involve your CDN or API gateway returning
cached responses if requests are overloading your subsystems. Or as
described above, your system might provide for an alternative path to
eventual consistency for specific failure modes. This is a far more
effective and customer focused approach than the presentation of a
generic error page that conveys ‘something has gone wrong’.

Resolve single points of failure

A single service can easily go from managing a single
responsibility of the product to multiple. For a startup, appending to
an existing service is often the simplest approach, as the
infrastructure and deployment path is already solved. However,
services can easily bloat and become a monolith, creating a point of
failure that can bring down many or all parts of the product. In cases
like this, you’ll need to understand ways to split up the architecture,
while also keeping the product as a whole functional.

At a fintech client, during a hyper-growth period, load
on their monolithic system would spike wildly. Due to the monolithic
nature, all of the functions were brought down simultaneously,
resulting in lost revenue and unhappy customers. The long-term
solution was to start splitting the monolith into several separate
services that could be scaled horizontally. In addition, they
introduced event queues, so transactions were never lost.

Implementing a microservice approach is not a simple and straightforward
task, and does take time and effort. Start by defining a domain that
requires a resiliency boost, and extract it’s capabilities piece by piece.
Roll out the new service, adjust infrastructure configuration as needed (increase
provisioned capacity, implement auto scaling, etc) and monitor it.
Ensure that the user journey hasn’t been affected, and resilience as
a whole has improved. Once stability is achieved, continue to iterate over
each capability in the domain. As noted in the client example, this is
also an opportunity to introduce architectural elements that help increase
the general resilience of your system. Event queues, circuit breakers, bulkheads and
anti-corruption layers are all useful architectural components that
increase the overall reliability of the system.

Regularly chaos test to validate system resilience

Chaos engineering is the bedrock of truly resilient products. The
core value is the ability to generate failure in ways that you might
never think of. And while that chaos is creating failures, running
through user scenarios at the same time helps to understand the user
experience. This can provide confidence that your system can withstand
unexpected chaos. At the same time, it identifies which user
experiences are impacted by system failures, giving context on what to
improve next.

Though you may feel more comfortable testing against a dev or QA
environment, the value of chaos testing comes from production or
production-like environments. The goal is to understand how resilient
the system is in the face of chaos. Early environments are (usually)
not provisioned with the same configurations found in production, thus
will not provide the confidence needed. Running a test like
this in production can be daunting, so make sure you have confidence in
your ability to restore service. This means the entire system can be
spun back up and data can be restored if needed, all through automation.

Start with small understandable scenarios that can give useful data.
As you gain experience and confidence, consider using your load/performance
tests to simulate users while you execute your chaos testing. Ensure teams and
stakeholders are aware that an experiment is about to be run, so they
are prepared to monitor (in case things go wrong). Frameworks like
Litmus or Gremlin can provide structure to chaos engineering. As
confidence and maturity in your resilience grows, you can start to run
experiments where teams are not alerted beforehand.

Recruit specialists with knowledge of resilience at scale

Hiring generalists when building and delivering an initial product
makes sense. Time and money are incredibly valuable, so having
generalists provides the flexibility to ensure you can get out to
market quickly and not eat away at the initial investment. However,
the teams have taken on more than they can handle and as your product
scales, what was once good enough is no longer the case. A slightly
unstable system that made it to market will continue to get more
unstable as you scale, because the skills required to manage it have
overtaken the skills of the existing team. In the same vein as
technical
debt,
this can be a slippery slope and if not addressed, the problem will
continue to compound.

To sustain the resilience of your product, you’ll need to recruit
for that expertise to focus on that capability. Experts bring in a
fresh view on the system in place, along with their ability to
identify gaps and areas for improvement. Their past experiences can
have a two-fold effect on the team, providing much needed guidance in
areas that sorely need it, and a further investment in the growth of
your employees.

Always maintain or improve your reliability

In 2021, the State of Devops report expanded the fifth key metric from availability to reliability.
Under operational performance, it asserts a product’s ability to
retain its promises. Resilience ties directly into this, as it’s a
key business capability that can ensure your reliability.
With many organizations pushing more frequently to production,
there needs to be assurances that reliability remains the same or gets better.

With your observability and monitoring in place, ensure what it
tells you matches what your service level objectives (SLOs) state. With every deployment to
production, the monitors should not deviate from what your SLAs
guarantee. Certain deployment structures, like blue/green or canary
(to some extent), can help to validate the changes before being
released to a wide audience. Running tests effectively in production
can increase confidence that your agreements haven’t swayed and
resilience has remained the same or better.