Building in Public #10: I Wouldn't Give This to My Friends Yet

Big picture: I want Vorx good enough to give to friends and family. We're not there yet.

That's the bar I'm working to. Not a number on a dashboard. Would I hand this to someone I know and not worry about what happens next? Not yet. But this week got us closer than any week so far, and I'll get to that at the end.

Three things are in the way

Build errors. Last week I published a post about build speed, about the 133 seconds of fixed cost sitting in front of every build. That measurement still looks right to me and I'm parking the work anyway. I want the build error rate close to zero before we get fancy about making things faster. Build errors first.

Previews. Getting better, definitely. Not there yet. I logged a few more bugs against them this week.

Updates. This is the worst of the three. Most apps a human builds do build. They take a while, but they get there. Ask for a change afterwards and it's a different story. The update system feels a lot worse than the build system right now. That's a problem, because nobody builds an app once and walks away. You build it and then you keep changing it.

We'd been measuring the wrong thing

I spent two days driving down the build failure rate and the biggest thing I found is that the rate itself was never real.

Eleven separate defects turned up in the measuring apparatus in 48 hours. Eight of them overstated failure. A check that judged nothing and reported a pass. A money check that only looked at the first table. A schedule check that couldn't see a date column that was sitting right there. Others understated it, marking builds green that had never actually rendered. Both biases running at the same time, in the same number, and no way to pull them apart. The 26.3% we'd been quoting came out of that.

So I rebuilt the five scenarios that kept failing, one at a time, against a build carrying every fix. Two of the failures weren't the product at all.

One was an allotment app scored BROKEN. Its database held exactly the 24 plots the brief asked for and the app served in two seconds with the right content. What actually failed was the preview being unreachable for the sixty seconds the scorer looked at it.

The other one is worse. A GAA club site got failed for rendering the same detail page for every team. I went and fetched the pages myself. /teams/u10 serves Kilmurry U10s, /teams/u14 serves Kilmurry U14s, different titles, different sizes, six differing lines. The app was fine. The gate was wrong, and it had failed that same scenario every round for days. Seven or eight repair rounds, up to five dollars a build, paying a model to fix an app that was already right.

The app was fine. The gate was wrong. We'd been paying a model to fix an app that was already right.

Plenty was genuinely broken underneath all that, and it's worth saying that almost none of it was the AI writing bad code. The template system was silently discarding the data file the generator had just written, so listings and detail pages were both reading from an empty array. The repair step was creating a second page route with the wrong name next to the correct one, so the checker and the site were looking at different files. The repair was manufacturing the defect it existed to fix, then spending the whole budget failing to reconcile it. Package installs were getting killed by the shared network storage. A gateway error page was crashing whole builds because a response got parsed as JSON before anyone checked it was JSON. And one production template had been pinned to an eighteen day old commit, so a fix that shipped two weeks earlier had never reached a single build.

All fixed and live now. The honest consequence is that the reported failure rate was probably going to go up, because two of those fixes deliberately turn silent passes into loud failures. A build that used to quietly hand someone a half empty app now stops instead. That's the number getting more honest, not the product getting worse. Doesn't look like progress on a chart though.

So here's the rule I'm keeping: when a build gets scored broken, check the boundary before you touch the code. Read the database. Fetch the page. Twice in one night what looked like a broken app was a broken ruler, and both times it cost real money before anyone checked.

It can't be me logging the bugs

That's the thing that's really bothering me this week.

I've got a fleet of AI agents shipping real work every day. And I'm still the one finding the bugs that matter, by using my own product and noticing when something is off. That won't scale, I can't be the bottleneck.

What I want is the system testing itself and finding its own problems. The tester should be going at the preview system automatically rather than waiting for me to open a page and spot something. And every so often something in there should stop and look at the big picture and the strategy instead of the next ticket.

There's a line I came across this week, from Sara Hooker, and it hasn't left me alone since: "The first step of any meaningful pursuit is to severely underestimate its difficulty."

I had a basic app working after a few days. I'm weeks in now, with a complicated stack, still solving build errors. I didn't get the app wrong. I got wrong how hard it is to know whether the app works.

In one day this week: an item marked shipped turned out to be half built, with the timer it was supposed to remove still running. A measurement tool we rely on had been reading 40 projects out of 259 and reporting it as a full week. A favicon upload silently throws your image away and tells you "saved". None of those are hard problems. All of them are invisible ones, and every one was costing us confidence in something we thought we already knew.

That's the bit you underestimate. Not the building. The knowing.

The bit that is ready

I set out to build 36 templates for launch. We've got 54. All built, tested and tweaked, and I think most of them are better than what the market leaders ship.

I couldn't help myself. Going the extra mile for people building apps is the whole idea behind Vorx, so I kept going.

This one is Harbourmaster, a supply chain simulator. I rejected the design about twelve times before I was happy with it.

Funny one. Snake, Tetris and 2048 showed up in the middle of my serious business templates and I was a bit WTF, because games are not what I'm trying to say Vorx is for. The AI patiently explained that I'd asked for them. Weeks back I'd made a throwaway comment about having something ready in case someone used Vorx to write a game. I meant a starting point. It built three finished games.

Turns out you have to be super careful with your words when you're talking to an AI.

None of them get in without being reviewed, and plenty do not get in at all.

Then I looked at what all this costs

Costs got completely out of hand this week.

There are two sides to this. Building the Vorx app itself, and building customer apps. On the customer side the big finding was that the repair loop was where essentially all of my cost was going, and that one type of template was causing most of the trouble. Same story as the section above, just arriving as a bill.

On our own side, the obvious move was to get off the expensive model. I shifted benchmarking, local testing and most of our code generation over to DeepSeek-Pro. For standard development work DeepSeek-Pro is about $0.90 per thousand coding requests against roughly $0.30 for DeepSeek-Flash. Clean 3x. I was delighted with myself for about a day.

I checked at five in the afternoon and it was $136.65 for the day. I wrote that down in my journal, noted it was only five o'clock, and carried on. By the end of the day it was $185.41. Eighteen thousand API requests. Five and a half billion tokens. In one day.

So the DeepSeek-Pro swap was a disaster and I rolled it back. Which is annoying, because it seemed like such an obviously good idea when I did it.

$9,392 of AI for about $100

Rolling back got me digging into what the expensive option had actually been costing, and that turned out to be the interesting part.

Claude stores its API costs locally, so I could go back and price work we'd already done as if we'd paid list rates. Seven days, one laptop. 33,940 calls across eight agents. $9,392 at API list rates. What I actually paid for that laptop was two subscriptions at $200 a month. Call it $100 for the week.

If my maths is mathing, that's about a 90x saving.

I didn't believe it, so I went and looked it up. The published figure is that the top subscription runs somewhere between 15x and 40x cheaper than buying the same tokens through the API. Mine is higher than that, so either we're unusually heavy users or I've got an error in there somewhere. Could be both. But the direction is clear enough.

33,940 calls across eight agents in seven days. $9,392 at list. I paid about $100.

So I did the opposite of what I'd done two days before. Stayed on Claude and added a third account.

Two good things came out of all that. Every single build call now gets logged to a central database, so I'm not finding out what a day cost me by logging into somebody else's billing page at eleven at night, which is genuinely how I found out about the $185. And the AI calls Vorx itself makes during testing and benchmarking are on the cheap model now instead of the good one.

Next on that list: DeepSeek reckon their new Flash model beats their own Pro model on performance, cost, speed and total time, at a 70 to 86% price reduction. If that holds it should be our default for testing and benchmarking. That's their claim though, not mine, so I want to check it against our own numbers first.

This whole product is a margin game. If I can get the systems for controlling margin in place from the start, we can be properly profitable and pass the savings on, rather than finding out about the economics when they're somebody else's problem to absorb.

So how is anyone making money on this

That 90x number bothered me enough that I went and asked Google how Anthropic could possibly be selling this. It came back with "your intuition is spot on", and said Anthropic is losing serious money on power users who max out plans like Claude Max 20x. It pointed me at a write-up of the plans and the API costs and at a thread arguing about exactly this, where the framing is the classic Silicon Valley subsidised land grab. The same playbook Uber and DoorDash and Amazon Prime ran. Buy market share and user loyalty with venture capital before somebody else does, worry about the unit economics later.

Which is fascinating. High stakes poker with real money.

My hunch is the land grab doesn't work, for what a hunch is worth. Developers will treat models as a commodity, because they are one, and they'll move to the next good-enough model at a lower price the week it shows up. I know that because it's exactly what I did four days ago, and then again two days after that.

I'll happily take advantage of the discounted AI rates their land grab is creating.

Running out of model

The other constraint I didn't see coming is that we keep running out of model.

We've got three Claude accounts across the company. Each one has a five hour usage limit, a weekly limit, and a separate weekly limit on the best model. The resets are staggered across the week rather than lined up, so working out who can do what and when has turned into a real scheduling job. Hitting a wall at four in the afternoon with two and a half hours to wait is very frustrating when the whole point of running a fleet is that it keeps going when I stop.

So there's one DeepSeek agent doing testing only, off-peak hours only, to take some load off those limits. DeepSeek is about half price off-peak, and off-peak is five hours on weekdays plus the whole weekend. Test builds don't care what time it is. Hmmm. Might be able to do a lot more with that.

Who knew running a software company is all about cost management. Then again, it always was.

And the agents keep stopping

Least interesting thing here, and it's near the top of my list anyway. The agents keep stopping. The loop each one runs is too long, so they sit idle a fair bit of the time. It's frustrating, and it's expensive, because I'm paying for capacity that isn't doing anything.

Where I've got to

Here's the good news, and it's the reason this week felt different.

The latest benchmark has build errors down to 12.9% over the last 48 hours. At one stage we were over 50%. That's the same number the whole first half of this post is about, and it moved because we fixed the ruler as well as the code.

It isn't where it needs to be. Our target is under 10% and I want it closer to zero than that before I start handing this to people I know. But 50% to 12.9% is a real direction and I'll take it.

The latest benchmark has build errors down to 12.9% over the last 48 hours. At one stage we were over 50%.

The bigger thing I've noticed is that I'm getting better at this. Better at directing a swarm of agents, better at taking myself out of the loop, better at keeping them pointed at the issues that matter, better at working at a level above the work. That's a skill I didn't have two months ago and I can feel it improving week to week.

So on one hand I'm frustrated this isn't going faster. On the other, I'm having the time of my life.

Onwards.