Running with the lights off.

A couple of weeks ago I wrote about the blast door: everything agents touch lives in a separate Cloudflare account, so even a buggy agent can’t reach production. This entry is about the machine on the safe side of that door.

On May 3rd I merged the first cut of an Elixir/OTP orchestrator that polls GitHub issues, spawns a Claude Code agent in a fresh git worktree for each one, streams its output, opens a PR against staging, and watches CI until it goes green. The first merge landed 64 files and 4,185 added lines, tests and docs included. The orchestrator that produced all of that is itself deliberately small: at its core a supervision tree you can hold in your head, one supervisor over a handful of processes, built to be dry, elegant, and modular the same way I hold the app code to. I’ve been calling it the dark factory, in the manufacturing sense: a dark factory is one automated enough that you can run it with the lights off, because there’s nobody on the floor who needs to see.

Where this came from

I didn’t arrive at this design on my own, and the honest version of the story has a false start in it.

The first plan wasn’t Elixir at all. I spent weeks designing this as a TypeScript harness, a port of Matt Pocock’s Sandcastle, and if you look at the factory’s module layout today (agent runner, stream buffer, prompt builder, session, hooks) you can still see Sandcastle’s skeleton in it. The shape survived. The language didn’t.

Three things I absorbed that spring changed the build. Stripe published their minions posts, and what stuck with me was the sandboxing: isolated environments cheap enough that you fan out five or ten agents per task and permission checks just disappear. StrongDM’s software-factory writing was the first of it I read, and it taught me how involved the process really is: the hard part isn’t generating code, it’s designing verification you can trust with the lights off. Their “Digital Twin Universe” idea (behavioral clones of your dependencies, so agents can test against the world without touching the real one) landed hard enough that the integration-test framework in this repo has been called DTU since early April. And then OpenAI open-sourced Symphony, their Codex orchestrator, a week before I merged mine. Ryan Lopopolo’s interview about it is the single best thing I’ve heard on running agents in production, and it connected a lot of dots for me. What I loved most was the integration design (the issue tracker as the control plane) and that they used AI to teach it: an agent writes the spec, a disconnected agent re-implements from the spec, and you loop until the spec reproduces the system. I wish I had that budget.

Symphony is Elixir/OTP. My factory is Elixir/OTP, merged six days after theirs went public. The language flip wasn’t one thing, it was a few landing in the same week. Theo Browne put out a video around then making the case that Elixir is one of the best languages to reach for with AI agents, and it lined up with what I was already feeling in the code: Elixir treats events and message-passing as first-class, which is exactly the shape of an orchestrator juggling many concurrent, long-running, crash-prone agents. Put that next to Symphony landing as a working reference in exactly that language, on top of the module shape I’d already sketched from Sandcastle, and the decision more or less made itself. The reasons it was right are below.

I also deviated from the advice on purpose. When I evaluated Symphony, the recommendation I got was to adopt Linear for task management and skip the daemon. I skipped both: no Linear, no new tool, GitHub issues and labels as the entire state machine. The reasoning is further down, and so far I haven’t regretted it.

Coding agents are a process-management problem

The thing I got wrong at first was thinking the hard part of running agents at volume would be the code they write. It isn’t. The code is usually fine. The hard part is everything around the writing.

An agent session is a child process, and like any process it can succeed, hang, crash, produce garbage, or quietly wander off-task while looking busy. To run more than one you need to spawn them, isolate them from each other, watch them, kill the ones that got stuck, and retry the ones that failed, all without losing the work that did complete. Written out like that, it stops sounding like an AI problem and starts sounding like an operating-systems problem.

It turns out there’s a whole platform whose founding premise is “processes fail, so supervise them,” and it isn’t TypeScript. It’s Erlang and its OTP framework. I first read about Erlang back when Facebook used it for their original chat backend, at my first 9-to-5 out of college at a B2B agency in Massachusetts (shout out iMarc), and it stuck with me: WhatsApp famously ran on it with a few dozen engineers serving hundreds of millions of users. The event-driven nature was the original reason I picked it here: the whole factory is a stream of events (an issue got labeled, an agent went quiet, CI went red), and this runtime treats that as the native shape of a program.

The reason I’d pick it again is one I didn’t fully appreciate going in: Elixir might be one of the better languages to pair with AI right now. It’s functional, so state doesn’t hide anywhere. Errors are explicit {:ok, result} and {:error, reason} values instead of surprises, and the compiler’s error messages are clear enough that an agent can usually read its own mistake and fix it. When the workers in your factory are language models, a language that fails loudly and legibly is worth a lot.

So the factory is Elixir, living in the same monorepo as the Workers code it builds, with a hard process-level boundary between the two. The Elixir side never imports any of the TypeScript; it only shells out to it, which keeps the orchestrator’s stability independent of the app’s.

The structure OTP gives you is a supervision tree: a hierarchy of processes where a supervisor watches its children and restarts them on a defined policy when they die. Ours is small and boring on purpose. A one_for_one supervisor (restart only the child that crashed, leave its siblings alone) sits over a config server, a task registry, a Task.Supervisor for the agent runs, and one orchestrator GenServer.

A GenServer is OTP’s standard stateful process, and this one is the brain: it traps exits so a dying agent notifies it rather than taking it down, and it drains running agents cleanly on shutdown instead of severing them mid-task. Each agent underneath it is an Erlang port, the runtime’s supervised wrapper around an external OS process, built around the claude CLI. The port’s output is consumed line by line and re-emitted as telemetry, so everything the agent says becomes an event the rest of the system can watch. A port that goes silent for 10 minutes gets a hard kill, on the assumption that a truly quiet agent is a stuck one.

Issue in, PR out

The whole interface is the issue tracker itself, which was a deliberate choice: I wanted the control surface to be something I already use, not a new dashboard to maintain.

Flag an issue for the factory and the orchestrator picks it up on its next 30-second poll, checks that its declared dependencies are already merged, and dispatches it. Dispatch means a new git worktree on its own branch (a worktree is a second checkout of the repo sharing one history, so parallel agents never fight over one working directory), a prompt assembled from a base template plus a per-task-type template (lint-fix, test-fix, and so on), and an agent turned loose inside that worktree.

On a clean completion the factory commits the work, pushes the branch, and opens the PR itself, targeting staging and labelled so the auto-merge contract from the last entry takes over from there. Then it polls CI every 30 seconds. A red build gets up to three repair attempts, where the agent is handed the failure and asked to fix it, before the issue is flagged as failed with a comment explaining what went wrong.

The task states live on the issue too, walking from picked-up through in-progress, PR-opened, and under-review to promoted, with a failed off-ramp. There’s no database anywhere in this. The issue tracker is the state machine, which has a property I like a lot: the factory’s entire memory is inspectable in the same UI I already live in, and if I want to know what it’s doing I look at the board, not at logs.

Knob	Value
Concurrent agents	3
Tracker poll	30s
Agent idle timeout	10 min
Agent total timeout	20 min
Retries	3, backoff 10s to 5 min

The first end-to-end run took issue #49 from label to merged fix as PR #55 with no human input. The first parallel run dispatched two issues at once, and both landed. The factory itself ships with 95 tests and runs under the same quality gates as everything else it builds, which felt important: an unattended tool that isn’t itself well-tested is just a faster way to make a mess.

The foreman

Autonomous doesn’t mean unattended, and I didn’t want to give up the ability to step in. The human control surface is called the foreman, and it does three things the raw claude CLI can’t.

pause kills the agent’s process but preserves its worktree and its session, so the work in progress survives. resume re-spawns the agent with that context intact and it picks up where it left off, which is what makes pausing safe rather than destructive. takeover is the one I reach for most: it pauses the task and drops me into an interactive session inside the agent’s own worktree, same conversation, same files, so I can steer for a few turns when it’s headed somewhere wrong and then hand control back.

There’s also a foreman-view, a tmux layout (tmux being a terminal multiplexer, one window split into many live panes) that tails the orchestrator log and auto-spawns a pane per running agent, with panes closing themselves as tasks finish. Watching three agents work in parallel from one terminal is about as close as this job gets to standing at a window over a factory floor.

Agent selection rides the same issue-tracker system: a different tag routes an issue to a different CLI agent entirely, so the factory isn’t married to a single model vendor and I can put a task in front of whichever agent suits it.

The factory’s first defect was procedural

On May 3rd the foreman PR merged itself to main instead of staging. Reverted within the hour, re-landed against the right branch, and that same day CI gained a branch guard: every PR must target staging, no exceptions. The point of the factory is to turn my mistakes into structural impossibilities. That has to include the mistakes the factory makes.

That incident is the whole philosophy in miniature, so it’s worth dwelling on. The fix wasn’t to review harder next time or to add a note to a checklist, because those decay. The fix was a deterministic gate: a branch guard that makes the wrong target literally unmergeable, after which that particular mistake can’t recur regardless of who or what opens the PR.

The factory’s PRs go through the identical pipeline mine do, staging first, then the soak timers and path denylists from the last entry, and promotion to production stays a human action with a biometric prompt at the end. That last part is the reassuring footnote to this whole incident: merging to main was the wrong branch, but it was never a path to a live store. Production takes my thumbprint, so the worst that bad merge could do was embarrass me, not reach a customer. The factory earns no shortcuts for being automated; if anything it’s held to the pipeline more strictly, because it can generate mistakes faster than I can.

What’s next in the log

The strange thing about getting the factory working is that it moved my bottleneck rather than removing it. Once the agents can reliably turn a well-specified issue into a merged, tested PR, the slow step becomes the specifying. My real job now sits upstream of all of this: writing plans good enough that Claude can take one and run with it, and getting them into the issue tracker faster than the factory drains the queue. The quality of the plan is the quality of the output, and that is where my hours go now.

The other limit is physical. “It’s just so much on my laptop. And I want it to continue running when my laptop is closed.” The supervision tree solves crashes, not geography. The factory still lives on one machine, which means it sleeps when I do, and the throughput I can get out of it is capped by my laptop being open. Moving it somewhere that doesn’t close at night is its own problem, with its own tradeoffs, and its own entry later on.

For now the factory is building the platform while I spend my attention on the product. That’s the trade I actually wanted from all this machinery: the factory makes me faster at shipping things merchants pay for because they make money, and the supervision, the gates, and the account boundary are what let me run it without lying awake. The next entry covers what that attention bought: server-side A/B testing at the edge, with the assignment decided before the page renders, so there’s nothing left on the client to flicker.