Monorepos

Tooling

Let's look at what tooling is required for a monorepo. I spoke to contacts at Facebook, Twitter and Google to understand how their tooling works, though lots of this information is also available in blogposts. I also have my own experience working in big organisations with monorepos.

Each of these big companies' monorepos consists of thousands of individual components living side-by-side in the same repository. An explicit dependency graph specifies what libraries depend on what. This dependency graph is used for build, test, and deployment processes. Indeed, monorepos go hand-in-hand with continuous integration and deployment tooling.

In a monorepo, you do not have freedom to pick versions of libraries: you're always picking the latest committed version. Therefore you have to ensure that those latest committed versions are the working versions - working together. A change to an individual component doesn't just have to pass the tests for that component: Everything that depends on it also has to pass tests. So monorepos are actually really big CI systems that run huge numbers of tests to ensure that nothing will break if a change is merged. If a change does break anything, anywhere in the monorepo, then it is rejected.

This may sound punishing: how can you make progress if a test can go wrong anywhere and prevent you committing? But remember, it's a single repo. So you can change an API, as long as you find and fix all uses of that API in the same commit.

Dependency hell

The primary driver for monorepos is to avoid dependency hell: the situation where different libraries argue about the versions of their dependencies they need.

A typical dependency hell situation is this: You depend on two libraries A and B. A and B both depend on C. If A and B need different versions of C, then you can't have them installed at the same time. Now, there may be a solution - a downgrade of A, an upgrade of B - but those both carry disadvantages.

Dependency hell is a significant problem, and one that becomes much more problematic as the organisation scales - as the number of libraries increases.

Monorepos avoid dependency hell by enforcing that A and B will always depend on the same version of C - the latest version.

Let's come back to dependency hell after we've looked at the rest of the monorepo landscape.

Up-to-date software

Another argument proposed for monorepos is to keep the whole organisation on similar versions of software, ideally as close as possible to the latest versions.

This seems to be a big selling point in some types of organisations. In the financial industry, there is a fear that running vastly different versions of software could, in some cases, create a situation that other parties in the market could exploit.

One problem is that nobody can quantify this. How much could this cost us? If it comes with a productivity cost for developers, is it worth paying? What about opportunity cost? What else are we trading off to get here? Reliability?

Making breaking API changes

I mentioned that monorepos are designed to prevent you committing changes that break anything - based on the success or failure of downstream unit tests.

The ideal is that you would develop as normal, but if you need to change an API, you simply go ahead and change all the uses of it.

In practice, this can't happen. Every component in a monorepo has its own development story... and remember, we're talking about very large developer organisations. Some developers are in the middle of complicated migrations. Some are on a demanding timescale to deliver some business critical feature. Some are on holiday. It's not workable to stomp over these people's constraints in order to ship your own feature. It's also not acceptable to just wait indefinitely to make your change.

So people compromise: they don't make breaking API changes, or make them behind feature flags. Or, they fork the component.

All of these outcomes are highly undesirable.

If you don't make breaking API changes, you're stuck with bad decisions forever. Your APIs grow crufty and painful to use - because you're locked into backwards-compatible mode.

By contrast, if you maintain different versions of a component, then you can easily make breaking changes: just release a new version. The previous version will continue to work just fine, and the consumer of a component chooses when to upgrade. You might maintain more than one major version at a time to ensure your users have time to upgrade.

Feature flags

Feature flags are just a configuration value that selects which version of an API to use. They enable backwards compatibility, because a caller that doesn't know about the feature flag just gets old behaviour. Code that elect to flip the flag can get new behaviour. The hope is that, in time, all the code can be updated to select the new behaviour.

if config.ENABLE_WOGGLE_PACKING:
    pack_all_the_woggles()
else:
    send_woggles_the_old_slow_way()

The problem with feature flags is that they are explicitly alternative codepaths. This multiplies the combinations of things that need to be tested. Beware also, that developers primarily care about the new behaviour. Are there still test sof the old behaviour or were they broken in the refactoring? Do they test the old behaviour in combination with new behaviours being added in other feature flags?

I see feature flags as equivalent to having different versions of software: but unlike versions of software, they aren't tracked, documented, and versioned. Developers have to understand them and reason about them themselves, and take pains to avoid breaking them. With versioned software, you reason about one version of software at a time, and a codepath that worked once will continue to work in that version.

Another problem is the longevity of feature flags. Stories abound of feature flags that introduced a new codepath, hoping to pursue a migration to a new, clean, API. And it never happens - both codepaths have to be maintained.

To return to the financial industry, Knight Capital found out first hand the problems of feature flags: they released software to a number of machines, one of which was configured to use an old feature flag, that activated an incorrect code path and lost the company $440 million.

So there could be a tangible effect on reliability of using feature flags.

Dealing with change

Monorepos change - a lot. Even though developers are shackled by a limited ability to make breaking API changes, the whole codebase is always changing and evolving. Developing against this is difficult in its own right.

While committed code cannot be broken due to the CI systems, developers may have to repeatedly merge their in-development changes with the changes from the rest of the monorepo in order to be able to commit them.

Well-managed versioned projects will publish useful changelog information with releases, which offers information to developers on what is changing. Developers can be informed about the gap between what they have been doing, and what they need to do from now on - and the version numbers tell you when. In my experience monorepo projects lack this: so even as you're likely to find things changing under you, you probably won't find information on what and why.

While developers might be able to deal with these issues, and merge incoming changes during their development cycle, QA practices like product owner review, UX testing or UAT require stability. Until a change has passed these processes, one has to hold off merging it - because then it can immediately be used by other users. However, holding off can mean that the world changes under you, and it's no longer mergeable without changes. Do these changes go back through the review process - in which case it's potentially never-ending - or do we just close our eyes to the fact that some unreviewed changes will slip out?

My answer is that these practices also need to be continuous in a monorepo ecosystem. Review and UAT and UX all have to happen incrementally all the time. The continuous nature of monorepos tends to force every other activity to be continuous.

• Integration with CI * Testing every change upstream and downstream * Developer productivity * Ability to run integration tests
• Non-standard VCS software
  • Inability to use off-the-shelf tooling
  • Cost of building and scaling your VCS tooling
  • Need to employ one size fits all solutions to avoid breaking your guarantees
• Ability to consume and produce open-source
• Big-ticket migrations such as Python 2-> 3
• Being able to make breaking API changes * Good enough is good enough * Stagnation * Forking
• Summary
  • Based on an antiquated view of software dependency - monoliths not microservices
  • How to do monorepos well if you still choose to do them

Dependency hell revisited

• Dependency hell revisited * Microservices * Microlibraries
• An alternative vision * Need to apply pressure to upgrade versions