Etsy, an online marketplace for unique, handmade, and vintage items, has
seen high growth over the last five years. Then the pandemic dramatically
changed shoppers’ habits, leading to more consumers shopping online. As a
result, the Etsy marketplace grew from 45.7 million buyers at the end of
2019 to 90.1 million buyers (97%) at the end of 2021 and from 2.5 to 5.3
million (112%) sellers in the same period.
The growth massively increased demand on the technical platform, scaling
traffic almost 3X overnight. And Etsy had signifcantly more customers for
whom it needed to continue delivering great experiences. To keep up with
that demand, they had to scale up infrastructure, product delivery, and
talent drastically. While the growth challenged teams, the business was never
bottlenecked. Etsy’s teams were able to deliver new and improved
functionality, and the marketplace continued to provide a excellent customer
experience. This article and the next form the story of Etsy’s scaling strategy.
Etsy’s foundational scaling work had started long before the pandemic. In
2017, Mike Fisher joined as CTO. Josh Silverman had recently joined as Etsy’s
CEO, and was establishing institutional discipline to usher in a period of
growth. Mike has a background in scaling high-growth companies, and along
with Martin Abbott wrote several books on the topic, including The Art of Scalability
and Scalability Rules.
Etsy relied on physical hardware in two data centers, presenting several
scaling challenges. With their expected growth, it was apparent that the
costs would ramp up quickly. It affected product teams’ agility as they had
to plan far in advance for capacity. In addition, the data centers were
based in one state, which represented an availability risk. It was clear
they needed to move onto the cloud quickly. After an assessment, Mike and
his team chose the Google Cloud Platform (GCP) as the cloud partner and
started to plan a program to move their
many systems onto the cloud.
While the cloud migration was happening, Etsy was growing its business and
its team. Mike identified the product delivery process as being another
potential scaling bottleneck. The autonomy afforded to product teams had
caused an issue: each team was delivering in different ways. Joining a team
meant learning a new set of practices, which was problematic as Etsy was
hiring many new people. In addition, they had noticed several product
initiatives that did not pay off as expected. These indicators led leadership
to re-evaluate the effectiveness of their product planning and delivery
processes.
Selecting a partner
Etsy had never worked with a software development consultancy. To be able
to scale product delivery and bring in some outside expertise, they started
looking for a vendor. After an extensive search, they selected Thoughtworks,
primarily because they perceived a close cultural fit and deep alignment
regarding modern software development.
Thoughtworks has a similar approach to Agile as Etsy – being Agile
rather than doing Agile. Etsy’s team was not looking to adopt a
methodology that blindly follows strict rituals and practices, and wanted
a partner that bases its approach on agile principles and culture.
Thoughtworks is known for technical excellence and leadership in DevOps,
Continuous Integration, and Continuous Delivery, all of which Etsy relies
on heavily. In addition, the two companies shared similar principles
regarding employee care, diversity, open-source, and technology
sustainability.
The Thoughtworks team started by embedding into product teams, working with
both buyer and seller teams, on various critical initiatives such as payment
model changes, Search Engine Optimization (SEO), and a notification platform.
This enabled Thoughtworks to understand how Etsy worked, created an immediate
impact on delivery, and improved development practices such as automated
testing.
For Thoughtworks, Etsy was quite different from the typical client.
They are a fully digital company, and their technical process and product
practises are very mature. Unlike most clients who engage Thoughtworks,
Etsy had no large transformation to undertake, and no ingrained habits
that needed a major shift. What they were seeking was outside perspective
and supplementive skills, to help them incrementally improve and tackle
their pressing initiatives. The culture and principles were already very
aligned with Thoughtworks’ own, so the partnership became more of an
exchange of ideas drawn from different experiences.
Scaling Product Delivery and Discovery
Thoughtworks and Etsy formed a cross-functional Product Delivery Culture
(PDC) team to analyze the current product delivery process. They looked across
the entire software development value stream and did a root cause analysis
into the complaints of managers and team members.
Thoughtworks and Etsy formed a cross-functional Product Delivery
Culture (PDC) team to analyze the current product delivery and discovery
process. They analyzed the entire software development value stream,
looking closely at the complaints of managers and team members. Selected
events were replayed to gain a deep understanding of “how” it happened
rather than who?. The Thoughtworks
team, because they hadn’t lived through the problems, were able to bring
an objective outsider perspective.
There were a number of things Etsy was doing really well:
- Cross-functional teams: They structured their teams around “4 table legs”:
Product, Design, Engineering, and Analytics. All planning and delivery practices
happen with collaboration among the groups. - Incremental delivery of value to users: Continuous delivery is the core
practice of their approach. A code change sitting in a repository and not
deployed to users is not valuable, and effectively costs the company money. Etsy
deploys hundreds of times a day, with a 30-minute commit to production lead
time. - Features are prioritized using data: Product managers and data analysts
collaborate to determine an expected value, measured by KPI changes. If the
feature does not deliver enough value versus its cost, as evidenced by an A/B
test, it is not rolled out, and the code is removed.
The team also uncovered opportunities for improvement:
- Each team was doing delivery in different ways, having disparate processes
and rituals, making onboarding new employees and changing teams difficult. Some
teams were using a process like Scrum, while others did not use any formal
process. - There were a number of initiatives that had not created value as
they expected. For example, Etsy tried a gift wrap feature that had
lower than expected adoption rates by sellers and buyers. - Prior to 2018, Etsy had been through a transitional period, with
pivots in product strategy and leadership changes leading some product
managers to expres a desire for greater ownership over their product
decisions.
Time to learning
The team created an improvement program, based on lean thinking and the
ideas of Marty Cagan, who had recently presented at Etsy. The leadership team was
reading his book Inspired.
I promise you that at least half the ideas
on your roadmap are not going to deliver what you hope. (By the way, the
really good teams assume that at least three quarters of the ideas won’t
perform like they hope.)
After some initial research the team came up with a metric they called Time
to Learning – the time it took for a product team to validate an idea with a
customer and gain a better understanding of its value. They had a baseline of
50 days that they wanted to reduce.
They were also looking at other metrics.
- Improving the time to market
- Improving the effectiveness of products (e.g. revenue, customer
engagement) - Employee happiness
To impact the KPIs, they came up with a number of solution hypotheses:
Light-weight prototypes that incorporate more direct user research
Etsy has strong experimentation infrastructure and analytics capabilities.
They run many concurrent A/B tests on the marketplace. The problem the PDC
team observed is that the feedback cycle can be quite slow. For an idea to be
ready for an A/B test it had to be near production-level quality. It also had
to have enough data to show statistical relevance. A/B tests would run for
months in lesser-used parts of the marketplace.
To set your expectations, strong teams normally
test many product ideas each week—on the order of 10 to 20 or more per week. I
want to emphasize that these are experiments, typically run using prototypes.
A prototype is not something that’s ready for prime time and certainly not
something your company would try to sell and stand behind. But they’re
immensely useful, as they’re all about learning fast and cheap.
To reduce the Time to Learning metric, the PDC team applied a lean UX
approach. First, they started by quickly creating lo-fi prototypes and
immediately showing them to users. Second, the team increased user feedback
sessions to every two weeks and reduced the level of ceremony. Third, they
created a dual-track system, with a design and research team continually
working ahead of a delivery team. The idea was to efficiently gain confidence
in an idea before it qualifies for an A/B test. By reducing the cost of
experimentation, they could increase the variety and quantity of experiments,
finding ideas that showed more value.
Figure 1: Experiment validation process
