Linking Modular Architecture to Development Teams

Recognising the Signs

Despite the best of intentions, software often deteriorates over time, both in
quality and performance. Features take longer to get to market, service outages
become more severe and take longer to resolve, with the frequent result that those
working on the product become frustrated and disenfranchised. Some of this can be
attributed to code and its maintenance. However, placing the blame solely on code
quality feels naive for what is a multifaceted issue. Deterioration tends to grow
over time through a complex interplay of product decisions, Conway’s law, technical
debt and stationary architecture.

At this point, it seems logical to introduce the organisation this article is based
around. Very much a large enterprise, this business had been experiencing a gradual
lengthening of the time it took to introduce new features into their retail
mobile application.

As a starter, the organisation had correctly attributed the friction they were
experiencing to increased complexity as their app grew- their existing development
team struggled to add features that remained coherent and consistent with the
existing functionality. Their initial reaction to this had been to ‘just add more
developers’; and this did work to a point for them. However, eventually it became
apparent that adding more people comes at the expense of more strained communication
as their technical leaders started to feel the increased coordination overhead.
Hence the ‘two
pizza’ rule promoted at Amazon: any team should be small enough to be fed by two
pizzas. The theory goes that by restricting how big a team can become, you avoid the
situation where communication management takes more time than actual value creation.
This is sound theory and has served Amazon well. However, when considering an
existing team that has simply grown too big, there is a tendency towards ‘cargo
culting’ Amazon’s example to try and ease that burden…

Limiting Cognitive Load

Indeed, the organisation was no exception to this rule: Their once small monolith had
become increasingly successful but was also unable to replicate the required rate of
success as it grew in features, responsibilities and team members. With looming
feature delivery deadlines and the prospect of multiple brand markets on the
horizon, they responded by splitting their existing teams into multiple smaller,
connected sub-squads – each team isolated, managing an individual market (despite
similar customer journeys).

This in fact, made things worse for them, as it shifted the communication tax from
their tech leadership to the actual team itself, while easing none of their
expanding contextual load. Realizing that communication and coordination was sapping
an increasing amount of time from those tasked with actual value creation, our
initial suggestion involved the idea of ‘cognitive
load
limitation’ outlined by Skelton & Pais (2019). This involves the
separation of teams across singular complex or complicated domains. These seams
inside software can be used to formulate the aforementioned ‘two pizza sized teams’
around. The result is much less overhead for each team: Motivation rises, the
mission statement is clearer, while communication and context switching are shrunk
down to a single shared focus. This was in theory a great solution to our client’s
problem, but can actually be misleading when considered in isolation. The benefits
from cognitive load limitation can only truly be realised if an application’s domain
boundaries are truly well defined and consistently respected inside the code.

Domain Driven Discipline

Domain
Driven
Design (DDD) is useful for organising complex logic into manageable groups
and defining a common language or model for each. However, breaking apart an
application into domains is only part of an ongoing process. Keeping tight control
of the
bounded context is as important as defining the domains themselves.
Examining our client’s application’s code we encountered the common trap of a clear
initial investment defining and organising domain responsibilities correctly, only
to have started to erode that discipline as the app grew. Anecdotal evidence from
stakeholders suggested that perpetually busy teams taking shortcuts driven by
urgent product
requirements had become the norm for the team. This in turn had contributed
to a progressive slowing of value delivery due to the accumulation of technical
debt. This was highlighted further still by a measurable downtrend in the
application’s Four
Key Metrics as it became more difficult to release code and harder to debug
issues.

Further warning signs of a poorly managed bounded context were discovered through
common code analysis tools. We found a codebase that had grown to become tightly
coupled and lacking in cohesion. Highly
coupled
code is difficult to change without affecting other parts of your system.
Code with low cohesion has many responsibilities and concerns that do not fit within
its remit, making it difficult to understand its purpose. Both these issues had been
exacerbated over time as the complexity of each domain inside our client’s app had
grown. Other indications came with reference again to cognitive load. Unclear
boundaries or dependencies between domains in the application meant that when a
change was made to one, it would likely involuntarily affect others. We noticed that
because of this, development teams needed knowledge of multiple domains to resolve
anything that might break, increasing cognitive load. For the organisation,
implementing rigorous control of each domain-bounded context was a progressive step
forward in ensuring knowledge and responsibility lay in the same place. This
resulted in a limitation of the ‘blast radius’ of any changes, both in the amount of
work and knowledge required. In addition, bringing in tighter controls in the
accruing and addressing of technical debt ensured that any short term
‘domain-bleeds’ could be rejected or rectified before they could grow

Another metric that was missing from the organisation’s mobile applications was optionality
of reuse. As mentioned earlier, there were multiple existing, mature brand
market applications. Feature parity across those applications was low and a
willingness to unify into a single mobile app was difficult due to a desire for
individual market autonomy. Tight coupling across the system had reduced the ability
to reuse domains elsewhere: Having to transplant most of an existing mobile app just
to reuse one domain in another market brought with it high integration and ongoing
management costs. Our utilisation of proper domain-bounded context control was a
good first step to modularity by discouraging direct dependencies on other domains.
But as we found out was not the only action we needed to take.

Domains that Transcend Apps

Scenario 1 – ‘The Tidy Monolith’

When viewed as a single application in
isolation, simply splitting the app into
domains, assigning a team, and managing their coupling (so as not to breach
their bounded contexts) works very well. Take the example of a feature request
to an individual application:

The
feature request is passed to the app squads that own the relevant domain. Our
strict
bounded context means that the blast radius of our change is contained within
itself, meaning our feature can be built, tested and even deployed without
having to
change another part of our application. We speed up our time to market and allow
multiple features to be developed simultaneously in isolation. Great!

Indeed, this worked well in a singular market context. However as soon as we
tried to address our second scaling problem- market feature disparity arising
from a lack of reusability – we started to run into problems.

Scenario 2 – ‘The Next Market Opportunity’

The next step for the organization on its quest for modularity of domains was to
achieve rapid development savings by transplanting parts of the ‘tidy monolith’
into an existing market application. This involved the creation of a common
framework (aspects of which we touch on later) that allowed
functionalities/domains to be reused in a mobile application outside its origin.
To better illustrate our methodology, the example below shows two market
applications, one in the UK, the other, a new app based out of the US. Our US
based application team has decided that in addition to their US specific domains
they would like to make use of both the Loyalty Points and Checkout domains as
part of their application and have imported them.

For the organisation, this appeared to mean an order of magnitude development
saving for their market teams vs their traditional behaviour of rewriting domain
functionality. However, this was not the end of the story- In our haste to move
towards modularity, we had failed to take into account the existing
communication structures of the organisation that ultimately dictated the
priority of work. Developing our previous example as a means to explain: After
using the domains in their own market the US team had an idea for a new feature
in one of their imported domains. They do not own or have the context of that
domain so they contact the UK application team and submit a feature request. The
UK team accepts the request and maintains that it sounds like “a great idea”,
only they’re currently “dealing with requests from UK based stakeholders”
so it’s unclear when they will be able to get to the work…

We found that this conflict of interest in prioritising domain functionality
limits the amount of reuse a consumer of shared functionality could expect –
this was evident with market teams becoming frustrated at the lack of progress
from imported domains. We theorized a number of solutions to the problem: The
consuming team could perhaps fork their own version of the domain and
orchestrate a team around it. However, as we knew already, learning/owning an
entire domain to add a small amount of functionality is inefficient, and
diverging also creates problems for any future sharing of upgrades or feature
parity between markets. Another option we looked into was contributions via pull
request. However this imposed its own cognitive load on the contributing team –
forcing them to work in a second codebase, while still depending on support on
cross team contributions from the primary domain team. For example, it was
unclear whether the domain team would have enough time between their own
market’s feature development to provide architectural guidance or PR reviews.

Scenario 3 – ‘Market Agnostic Domains’

Clearly the problem lay with how our teams were organised. Conway’s
law is the observation that an organisation will design its business
systems to mirror its own communication structure. Our previous examples
describe a scenario whereby functionality is, from a technical standpoint
modularised,
however
from an
ownership standpoint is still monolithic: “Loyalty Points was created
originally
for the UK application so it belongs to that team”. One potential
response to this is described in the Inverse
Conway Maneuver. This involves altering the structure of development teams
so that they enable the chosen technical architecture to emerge.

In the below example we advance from our previous scenario and make the
structural changes to our teams to mirror the modular architecture we had
previously. Domains are abstracted from a specific mobile app and instead are
autonomous development teams themselves. When we did this, we noticed
relationships changed between the app teams as they no longer had a dependency
on functionality between markets. In their place we found new relationships
forming that were better described in terms of consumer and provider. Our domain
teams provided the functionality to their market customers who in turn consumed
them and fed back new feature requests to better develop the domain product.

The main advantage this restructuring has over our previous iteration is the
clarification of focus. Earlier we described a conflict of interest that
occurred when a market made a request to change a domain originating from within
another market. Abstracting a domain from its market changed the focus from
building any functionality solely for the benefit of the market, to a more
holistic mission of building functionality that meets the needs of its
consumers. Success became measured both in consumer uptake and how it was
received by the end user. Any new functionality was reviewed solely on the
amount of value it brought to the domain and its consumers overall.

Focus on Developer Experience to Support Modularity

Recapping, the organisation now had a topological structure that supported modularity
of components across markets. Autonomous teams were assigned domains to own and
develop. Market apps were simplified to configuration containers. In concept, this
all makes sense – we can plot how feedback flows from consumer to provider quite
easily. We can also make high level utopian assumptions like: “All domains are
independently developed/deployed” or “Consumers
‘just’ pull in whatever reusable domains they wish to form an application”.

In practice,
however, we found that these are difficult technical problems to solve. For example,
how
do you maintain a level of UX/brand consistency across autonomous domain teams? How
do
you enable mobile app development when you are only responsible for part of an
overall
application? How do you allow discoverability of domains? Testability? Compatibility
across markets? Solving these problems is entirely possible, but imposes its own
cognitive load, a responsibility that in our current structure did not have any
clear
owner. So we made one!

The Results

We’ve told the story about how we modularised a mobile app, but how successful was it
over time? Obtaining empirical evidence can be difficult. In our experience, having
a legacy app and a newly architected app within the same organisation using the same
domains with delivery metrics for both is a scenario that doesn’t come around too
often. However luckily for us in this instance, the organisation was large enough to
be transitioning one application at a time. For these results, we compare two
functionally similar retail apps. One legacy with high coupling and low cohesion
albeit with a highly productive and mature development team (“Legacy monolith”). The
other, the result of the modular refactoring exercise we described previously – a
well defined and managed bounded context but with ‘newer’ individual domain teams
supporting (“Domain-bounded Context App”). Cycle time is a good measure here
as it represents the time taken to ‘make’ a change in the code and excludes pushing
an app to the store- A variable length process that App type has no bearing on.

Even when cycle time was averaged across all domain teams in our second app we saw a
significant uplift versus the Legacy App with a less experienced team.

Our second comparison concerns optionality of re-use, or lack thereof. In this
scenario we examine the same two mobile apps in the organisation. Again, we compare
one requiring existing domain functionality (with no choice but to write it
themselves) with our modular app (able to plug and play an existing domain). We
ignore the common steps on the path to production since they have no impact on what
we are measuring. Instead, we focus on the aspects within the control of the
development team and measure our development process from pre-production ‘product
sign off’ to dev-complete for a single development pair working with a designer
full-time.

The dramatically different figures above show the power of a modular architecture in
a setting that has a business need for it.

As an aside, it is worth mentioning that these external factors we have excluded
should also be measured. Optimising your development performance may reveal other
bottlenecks in your overall process. For example, if it takes 6 months to create a
release, and governance takes 1 month to approve, then governance is a comparatively
small part of the process. But if the development timeline can be improved to 5
days, and it still takes 1 month to approve, then compliance
may become the next bottleneck to optimise.

One other advantage not represented in the results above is the effect a team
organised around a domain has on integration activities. We found autonomous
domain teams naturally seconding themselves into market application teams in an
attempt to expedite the activity. This, we believe, stems from the shift in focus of
a domain squad whereby success of its domain product is derived from its adoption.

We discovered two concentric feedback loops which impact the rate of adoption. The
outer, a good integration experience from the consumer of the domain (i.e. the app
container). This is a developer-centric feedback loop, measured by how easily the
consumer could configure and implement the domain as part of their overall
brand-specific product offering. The inner, a good end user experience – how well
the overall journey (including the integrated domain) is received by the consumer’s
market customer. A poor consumer experience impacts adoption and ultimately risks
insulating the domain team from the actual users of the capability. We found that
domain teams which collaborate closely with consumer teams, and which have direct
access to the end users have the fastest feedback loops and consequently were the
most successful.

The final comparison worth mentioning is one derived from our Platform domain.
Starting a new piece of domain functionality is a time consuming activity and adds
to the overall development cost for functionality. As mentioned earlier, the
platform team aims to reduce this time by identifying the pain points in the process
and optimising them – improving the developer experience. When we applied this model
to domain teams within our modular architecture we found an over 80% reduction in
startup costs per team. A pair could achieve in an afternoon activities that had
been estimated for the first week of team development!

Limitations

By now you should have quite a rosy picture of the benefits of a modular architecture
on mobile. But before taking a sledgehammer to your ailing monolithic app, it’s
worth bearing in mind the limitations of these approaches. Firstly, and indeed most
importantly, an architectural shift such as this takes a lot of ongoing time and
effort. It should only be used to solve serious existing business problems
around speed to market. Secondly, giving autonomy to domain teams can be both a
blessing and a curse. Our platform squad can provide common implementations in the
form of sensible defaults but ultimately the choices are with the teams themselves.
Naturally, coalescing on platform requirements such as common UI/UX is in the
interest of the domain squads if they wish to be incorporated/accepted into a market
app. However, managing bloat from similar internal dependencies or eclectic
design
patterns is tricky. Ignoring this problem and allowing the overall app to
grow uncontrolled is a recipe for poor performance in the hands of the customer.
Again, we found that investment in technical leadership, in conjunction with robust
guardrails and guidelines helps to mitigate this problem by providing
architecture/design oversight, guidance and above all communication.

Summary

To recap, at the start of this article we identified two significant delivery
problems exhibited in an organisation with a multi app strategy. A lengthening of
the time it took to introduce new features into production and an increasing
feature
disparity between other similar in house applications. We demonstrated that
the solution to these problems lies not in a single strategy around technical
architecture, team structure or technical debt, but in a simultaneously evolving
composite of all those aspects. We started by demonstrating how evolving team
structures to support the desired modular and domain-centric architecture improves
cognitive and contextual load, while affording teams the autonomy to develop
independently of others. We showed how a natural progression to this was the
elevation of teams and domains to be agnostic of their originating
application/market, and how this mitigated the effects of Conway’s law inherent with
an application monolith. We observed that this change allowed a consumer/provider
relationship to naturally occur. The final synchronous shift we undertook was the
identification and investment in the ‘platform’ domain to solve central problems
that we observed as a consequence of decoupling teams and domains.

Putting all these aspects together, we were able to demonstrate a 60% reduction in
cycle time averaged across all modular domains in a market application. We also
saw an 18 fold improvement in development cost when integrating modular
domains to a market app rather than writing from scratch. Furthermore, the focus on
engineering effectiveness allowed our modular architecture to flourish due to the 80%
reduction
in startup costs for new domains and the ongoing support the ‘platform team’
provided. In real-terms for our client, these savings meant being able to capitalise
on market opportunities that were previously considered far too low in ROI to
justify the effort – opportunities that for years had been the uncontested domains
of their competitors.

The key takeaway is that a modular architecture intrinsically linked to teams can be
highly beneficial to an organisation under the right circumstances. While the
results from our time with the highlighted organisation were excellent, they were
specific to this individual case. Take time to understand your own landscape, look
for the signs and antipatterns before taking action. In addition, do not
underestimate the upfront and ongoing effort it takes to bring an ecosystem like
that which we have described together. An ill considered effort will more than
likely cause more problems than it solves. But, by accepting that your situation
will be unique in scope and thus resisting the pull of the ‘cargo cult’: Focusing on
empathy, autonomy and lines of communication that enable the architecture at the
same time, then there is every reason you could replicate the successes we have
seen.