How A Cabin In The Woods Proved Better, Faster, and Cheaper Than Microsoft's Datacenters

After years of using GitHub, we decided to self-host. Here is the scoreboard.

GitHubGitea
Speed6.7 merges / day15.4 merges / day
Price~$30,000 / year$10,000 and a week of work once
Uptimegithubstatus.com1 outage, fixed in 13 minutes
Machinesslow, shared, flakyours, and fast
Queueingbehind one shared stacka leased slot per job
Sovereigntyless than none100% ours

One of the largest technology companies on earth lost on price, speed, and uptime to a single 15U rack in the woods, and the capital cost of beating it was a third of what we were paying it every year. The running costs round to nothing: the power is solar, and the uplink is about $100 a month. This is a surprising outcome and it runs completely counter to startup wisdom. Startups almost always buy rather than build, because any time spent on anything other than the product is usually an unaffordable opportunity cost.

A Microsoft datacenter cold aisle The 15U rack

Every test you write is a tax on the CI queue, and on a slow queue that tax compounds until the safety net is the bottleneck. Owning the runners removed the tax instead of the tests. That is the actual argument for self-hosting, and it has almost nothing to do with the sticker price.

The point of GitHub is to manage code repositories. This includes CI, the practice of running tests that must pass before changes to the repository are merged (and ultimately shipped to end users). These tests keep development speed and software quality from entering into conflict with each other. Speed is the number of changes made per unit time, but each change can introduce bugs that can range from minor to catastrophic. Testing helps prevent bugs from ever deploying and makes it much easier to find them when they arise. If a test fails, the bug is probably somewhere in the set of changes being submitted and there’s no need to scour the entire repository. It’s the difference between combing through the last few hours of work and potentially years. As long as you have good test coverage, you can make changes frequently, test them, and ship them. The faster that cycle, the faster the product is built and the more value you can deliver to users over time. There’s a reason it’s a classic recipe for success. It works.

However, if CI takes too long, you end up waiting. When I was a kid, you waited for your code to compile. Then, if you were doing machine learning, you had to wait for your models to train. And then for your agents to finish. But in the last few years, GitHub’s deteriorating performance made CI the least glamorous wait of them all, like a Soviet breadline where you reach the front only to find there’s no more bread. At first we paid a ridiculous premium for bigger machines, but eventually it didn’t matter because the service was down too frequently.

It’s hard to imagine any self-respecting developer being a GitHub apologist at this point, but if there are any out there, let’s just get this out of the way: sure, we could have tweaked the CI YAML (and we did). Sure, we could have optimized our tests (and we did). Sure, we could have parallelized the tests and deployed more runners and paid for more extra-large instances (and we did). But at a certain point, it was going to be less effort and less cost to get a faster and more reliable result by deploying Gitea (a great open-source alternative to GitHub) and rolling our own CI/CD runners (and we did).

What The $30,000 To GitHub Was Actually Buying

A queue slow enough to set the pace of the entire project. It took about a week to really get our new forge going. From there on out, the number of PRs a day more than doubled. Put another way, GitHub was more than halving our productivity by making us wait every day, all day, in the tech equivalent of a long line at the DMV staffed exclusively by the Zootopia sloths. The next best alternative we could think of was to start cutting tests, which is a bit like cutting out swaths of the hull on a sinking ship to reduce the ship’s weight so it doesn’t sink as fast. And the thought of adding new tests made me wince… not exactly the incentives you want when building software that has to be robust.

So we were paying $30,000 a year for a standing incentive to cancel out some of the most important engineering work we do.

Our Thesis On Display

We are building governance software on the thesis that centralized institutions are failing to process the complexity of the systems they administer, and that the cost of their failure (be it moral, technical, or financial) is externalized onto everyone downstream. Self-hosting the forge is that same argument turned on ourselves: we did not want to hand custody of our data and the pace of our own work to a company whose incentive is for our queue to stay exactly as slow as we will tolerate.

The fact that the sovereign option also turned out to be the cheaper and faster one is noteworthy. The conventional wisdom that owning your infrastructure is a luxury for people with time to burn was, in this case, backwards in all three dimensions at once. That is what the incentives of a centralized provider produce given enough time: Bugatti prices for Trabant quality, on a ten-year waiting list, because nothing in the arrangement pays them to be any better than the least you will put up with.