I Built 17 Automations. Five Survived the Year.
A post-mortem on the twelve that died. The pattern behind the survivors turned out to be embarrassingly simple, and it isn't about the tooling.
Over about two years I built seventeen automations. Five are still running. This is a post-mortem on the twelve that died, because the pattern behind the survivors turned out to be embarrassingly simple — and it has nothing to do with the tools.
The graveyard
The twelve that died, grouped by how they died:
Died because the source changed (5)
Scrapers, mostly, plus two integrations that depended on a page layout. Something upstream changed, the automation broke silently, and I found out days or weeks later. Silent failure is worse than no automation, because you keep trusting output that stopped arriving.
Died because I stopped trusting them (4)
These technically still worked. But each produced output I had to check anyway, and once you're checking everything the automation has become a slower way of doing the task. I killed these with some reluctance and considerable relief.
Died because the task went away (2)
Fine. Nothing wrong with these — the underlying need disappeared. Worth noting only because I'd spent a full day building one of them for a process that existed for four months.
Died because maintaining it cost more than doing it (1)
A genuinely clever multi-step thing that broke every few weeks. I maintained it for eight months out of pure sunk-cost pride. Doing the task by hand takes about six minutes a week.
The five survivors
- Nightly encrypted backup of everything that matters, with a weekly restore test. Runs unattended, tells me loudly when it fails.
- Invoice generation from a time log. Deterministic input, deterministic output, and an error is visible immediately because I read every invoice before sending it.
- Inbound triage — sorting and tagging, no actions taken. If it tags something wrong the cost is that I see a message in the wrong folder.
- Site deploy on git push. Fails loudly, fails immediately, and the output is checkable by loading the page.
- Weekly metrics digest pulled from three sources into one email. If a number looks wrong I can check it in about a minute.
The pattern
Every survivor has all three of these. Every casualty was missing at least one.
- It fails loudly. When it breaks, I know that day. Silent failure killed five of the twelve on its own.
- The output is cheap to verify. Not "doesn't need verification" — cheap. A glance, not an investigation. This is the one that eliminated the four I stopped trusting.
- The input is stable. It doesn't depend on someone else's page layout, undocumented API, or a format that changes when a vendor feels like it.
Notice that none of these are about the automation being clever, or about which tool built it. The tooling was essentially irrelevant to survival. I built things in four different platforms and the death rate was the same in all of them.
Three properties. Every automation I still run has all three. Every one I killed was missing at least one. I now check these before building rather than after.
The rule I use now
Before building anything, I answer three questions in writing:
- How will I find out when this breaks — and does that mechanism itself depend on the thing working?
- How long does checking one output take? If it's more than about 10% of doing the task manually, I don't build it.
- What does this depend on that I don't control, and how often has that changed historically?
This takes about five minutes and it has stopped me building maybe six things in the last year. Given the historical death rate, I count that as the most productive five minutes in the process.
The uncomfortable conclusion, after all of it: most of my automation time was spent building things that didn't survive. The value came from a handful of dull, robust, loudly-failing scripts — and I'd have got to them years earlier if I'd started with the three properties instead of the tools.
The five-minute test, worked through
Here's the test applied to something I nearly built last month: automatically drafting replies to inbound course-support emails.
How will I find out when it breaks? Badly. A drafting automation fails by producing plausible, wrong drafts — not by stopping. There's no error to catch. Already a serious warning sign.
How long does checking one output take? To check a support reply properly I have to read the original email and the draft, and understand the account context. That's roughly 80% of just writing the reply myself. Fails the 10% threshold by a wide margin.
What does it depend on that I don't control? The support platform's API, plus the model's behaviour. Both have changed under me before.
Three failures out of three. I didn't build it, and the five minutes spent working through this saved me what I'd conservatively estimate at a day of building plus several months of low-grade maintenance.
Compare the nightly backup, which I did build: it fails loudly (no completion notification by 6am, I get an alert), the output is checkable in seconds (does the restore test pass), and it depends only on my own disk and a storage bucket. Three for three the other way.
Designing for loud failure
Silent failure killed five of my twelve casualties, so it's worth being concrete about how to prevent it. Three techniques, in increasing order of usefulness.
Alert on error. The obvious one, and the weakest. It only catches failures that throw. A scraper returning an empty result set because the page layout changed doesn't throw — it succeeds, emptily, forever.
Alert on absence. Better. Expect the thing to report success by a certain time; alert if it doesn't. This catches the case where the automation didn't run at all, which is the failure mode monitoring most often misses because a dead process sends no errors.
Alert on implausibility. The one that actually works. Define what a normal result looks like and alert when reality falls outside it. My backup alerts if the archive is more than 20% smaller than last week's. My metrics digest flags any number that moved more than 40% week-on-week. Neither is clever, and between them they've caught every silent failure I've had since 2024.
The general principle: monitor the output's shape, not just the process's exit code. Processes lie by succeeding. Output shape is much harder to fake.
Questions I get about this
Found an error in this piece? Tell me — corrections get made in the open with a dated note, never a silent edit. The log is public.
If you want the long version: there's a course — AI Systems for Solo Operators, $297, one payment. If this post already solved your problem, you don't need it.