Introducing a product delivery culture at Etsy

Selecting a partner

Etsy had never worked with a software development consultancy. To be able
to scale product delivery and bring in some outside expertise, they started
looking for a vendor. After an extensive search, they selected Thoughtworks,
primarily because they perceived a close cultural fit and deep alignment
regarding modern software development.

Thoughtworks has a similar approach to Agile as Etsy – being Agile
rather than doing Agile. Etsy’s team was not looking to adopt a
methodology that blindly follows strict rituals and practices, and wanted
a partner that bases its approach on agile principles and culture.
Thoughtworks is known for technical excellence and leadership in DevOps,
Continuous Integration, and Continuous Delivery, all of which Etsy relies
on heavily. In addition, the two companies shared similar principles
regarding employee care, diversity, open-source, and technology
sustainability.

The Thoughtworks team started by embedding into product teams, working with
both buyer and seller teams, on various critical initiatives such as payment
model changes, Search Engine Optimization (SEO), and a notification platform.
This enabled Thoughtworks to understand how Etsy worked, created an immediate
impact on delivery, and improved development practices such as automated
testing.

For Thoughtworks, Etsy was quite different from the typical client.
They are a fully digital company, and their technical process and product
practises are very mature. Unlike most clients who engage Thoughtworks,
Etsy had no large transformation to undertake, and no ingrained habits
that needed a major shift. What they were seeking was outside perspective
and supplementive skills, to help them incrementally improve and tackle
their pressing initiatives. The culture and principles were already very
aligned with Thoughtworks’ own, so the partnership became more of an
exchange of ideas drawn from different experiences.

Scaling Product Delivery and Discovery

Thoughtworks and Etsy formed a cross-functional Product Delivery
Culture (PDC) team to analyze the current product delivery and discovery
process. They analyzed the entire software development value stream,
looking closely at the complaints of managers and team members. Selected
events were replayed to gain a deep understanding of “how” it happened
rather than who?. The Thoughtworks
team, because they hadn’t lived through the problems, were able to bring
an objective outsider perspective.

There were a number of things Etsy was doing really well:

• Cross-functional teams: They structured their teams around “4 table legs”:
  Product, Design, Engineering, and Analytics. All planning and delivery practices
  happen with collaboration among the groups.
• Incremental delivery of value to users: Continuous delivery is the core
  practice of their approach. A code change sitting in a repository and not
  deployed to users is not valuable, and effectively costs the company money. Etsy
  deploys hundreds of times a day, with a 30-minute commit to production lead
  time.
• Features are prioritized using data: Product managers and data analysts
  collaborate to determine an expected value, measured by KPI changes. If the
  feature does not deliver enough value versus its cost, as evidenced by an A/B
  test, it is not rolled out, and the code is removed.

The team also uncovered opportunities for improvement:

• Each team was doing delivery in different ways, having disparate processes
  and rituals, making onboarding new employees and changing teams difficult. Some
  teams were using a process like Scrum, while others did not use any formal
  process.
• There were a number of initiatives that had not created value as
  they expected. For example, Etsy tried a gift wrap feature that had
  lower than expected adoption rates by sellers and buyers.
• Prior to 2018, Etsy had been through a transitional period, with
  pivots in product strategy and leadership changes leading some product
  managers to expres a desire for greater ownership over their product
  decisions.

Time to learning

The team created an improvement program, based on lean thinking and the
ideas of Marty Cagan, who had recently presented at Etsy. The leadership team was
reading his book Inspired.

I promise you that at least half the ideas
on your roadmap are not going to deliver what you hope. (By the way, the
really good teams assume that at least three quarters of the ideas won’t
perform like they hope.)

— Marty Cagan, Inspired (p19)

After some initial research the team came up with a metric they called Time
to Learning – the time it took for a product team to validate an idea with a
customer and gain a better understanding of its value. They had a baseline of
50 days that they wanted to reduce.

They were also looking at other metrics.

• Improving the time to market
• Improving the effectiveness of products (e.g. revenue, customer
  engagement)
• Employee happiness

To impact the KPIs, they came up with a number of solution hypotheses:

Light-weight prototypes that incorporate more direct user research

Etsy has strong experimentation infrastructure and analytics capabilities.
They run many concurrent A/B tests on the marketplace. The problem the PDC
team observed is that the feedback cycle can be quite slow. For an idea to be
ready for an A/B test it had to be near production-level quality. It also had
to have enough data to show statistical relevance. A/B tests would run for
months in lesser-used parts of the marketplace.

To set your expectations, strong teams normally
test many product ideas each week—on the order of 10 to 20 or more per week. I
want to emphasize that these are experiments, typically run using prototypes.
A prototype is not something that’s ready for prime time and certainly not
something your company would try to sell and stand behind. But they’re
immensely useful, as they’re all about learning fast and cheap.

— Marty Cagan, Inspired (p27)

To reduce the Time to Learning metric, the PDC team applied a lean UX
approach. First, they started by quickly creating lo-fi prototypes and
immediately showing them to users. Second, the team increased user feedback
sessions to every two weeks and reduced the level of ceremony. Third, they
created a dual-track system, with a design and research team continually
working ahead of a delivery team. The idea was to efficiently gain confidence
in an idea before it qualifies for an A/B test. By reducing the cost of
experimentation, they could increase the variety and quantity of experiments,
finding ideas that showed more value.

Figure 1: Experiment validation process

Blueprint for product delivery

The PDC, in conjunction with product team leaders, came up with a set of
product delivery principles and related delivery practice changes. The
practices included: smaller more focused stories, redesigning story walls to
better fit the team process, using spikes to reduce technical uncertainty, and
better ways of managing tech debt. They did not want to remove the autonomy of
teams. Continuous improvement is a big part of Etsy culture. These practices
are meant to be treated as sensible defaults that can be adapted and improved
for each team’s specific context.

The PDC team first tested the practice changes on pilot product teams to
establish their effectiveness and then iterated. Once the practice
demonstrated an improvement, the team added to a knowledge base and provided
light coaching to introduce the new practices to all product teams across
Etsy.

Lean Portfolio management

Some large initiatives were still being carried out, even though there was
not clear evidence of the initiative creating value. This is the typical sunk
cost problem. To solve this, the PDC team
suggested taking a lean approach to portfolio management.

Our runway should be a list of hypotheses to test, not a list of
requirements to build. When we reward our teams for their ability to deliver
requirements, it’s easy to rapidly bloat our products with unnecessary
features—leading to increased complexity, higher maintenance costs, and
limited ability to change. Features delivered are not a measure of success,
business outcomes are. Our runway is a series of questions that we need to
test to reduce uncertainty and improve our understanding of growth
opportunities.

— Humble et al, Lean Enterprise (p96)

Instead of creating lengthy project specifications and plans, the PDC team
coached product teams to develop outcome statements for each initiative. With
outcome statements, starting with a customer or business outcome allows the
team to experiment with many different ideas to achieve the outcome by tying
them to measurable KPIs. The less precise an outcome is, including its KPIs,
the harder it is to specify. When preparing for possible initiatives, each
product team created a two-page brief summarizing the outcome, the measurable
impact, and the initial investment required.

The executive team prioritized the outcome statements, and each statement
was viewed as a “bet” amongst a portfolio. Inspired by the Lean Value Tree
concept, the executive team would practice a regular cadence to determine if a
bet was paying off, giving them the opportunity to pivot early.

Stronger product & engineering collaboration

Another improvement was to include engineers early in the product ideation
process and to have them join user feedback sessions. The team tested
engineers joining the hypothesis generation and validation sessions and
working closely with designers during design sprints. The results of the test
showed a number of benefits. Engineers improved their understanding of the
customer, building empathy for their needs. They could provide on-the-spot
feedback on the feasibility of an approach and quickly create prototypes to
demonstrate ideas to users. Cross-functional engagement allowed engineers to
contribute a number of great ideas that made it to the product.

After the initial PDC initiative Etsy continued to strengthen the
collaboration between groups. At every level of the organization, product and
engineering are partners, which means they’re able to have healthy trade-off
discussions. The platform group expanded the use of product managers for
infrastructure and developer tooling teams – encouraging teams to think about
technical capabilities as products and end-users being Etsy engineers. The
technical product managers do research and prioritization activities, just as
they would for a customer product.

What was the impact of product delivery culture?

Testing with pilot teams indicated that the Time to Learning went from 50
days to 5 days. During the pilot, one product manager said, “What I love about
the [direct user research sessions] pilot is that it lets us test our ideas
faster, it was really nice to not spend as much time diving deep into
something that we didn’t even know we were going to use.” An initiative on
inventory quality that was predicted to deliver a seven million dollar revenue
increase instead delivered eleven million dollars in incremental value using
the PDC suggested improvements. Following this pilot success, the team coached
the entire product development organization on the product delivery approach.

According to Mike Fisher, “It is very hard to isolate the benefits of a
process change on an organization of several hundred engineers since we don’t
run A/B experiments of this type. However, we did look at high level
productivity metrics after having PDC implemented for over 2 years, and all of
our metrics were very positive. We had improved our velocity by doing this
work, and we were able to produce ultimately more benefits for the buyers and
sellers – more features, better search, better processes; the whole
marketplace became better for buyers and sellers and that was the goal.”