Building in Public #11: One Hell of a Frustrating Week

It's been one hell of a frustrating week, to be honest. And it blew by.

Here is the blow by blow.

Monday: I am paying a genius to do a school child's job

I had told my build expert to use a cheap model for our benchmark rounds, and it did exactly that. What I had not spotted is that when a build goes wrong there is a repair step, and the repair step was quietly escalating to the most expensive model we have. Every time.

Over a hundred and fifty dollars a day, on retries.

Fixed with a hard budget and proper reporting. And every AI call we make now, whether it is us building Vorx or Vorx building somebody's app, on whatever model, lands in one central database I can slice any way I like. I can finally see where the money goes instead of finding out later.

Tuesday: an agent ground the whole fleet to a halt

Just as I walked out the door to collect the kids from school.

One of my agents was tidying up after itself and ran a command to kill its own leftover processes. It got the arguments in the wrong order. On a Mac that means the command stops reading them as instructions and starts treating them as things to match, and one of them was the number 1.

Every agent I have is called Lap1-something. My database runs out of a folder with 18 in the name.

It killed everything. We replayed it safely afterwards and the command it ran selected 356 of the 931 things running on that laptop. The one it meant to run selected zero. My database was down for an hour and twenty five minutes and I was in a school car park watching it happen on my phone.

Here is the bit I am pleased about. We did not write a rule telling everyone to be careful. There was already a rule. Instead the machine now physically refuses that command before it can run. It was live the same evening. Then we went back through the last seven days of everything the fleet had typed, all 52,875 commands, and found 42 more that could have done the same thing.

A rule is a note. A machine that says no is a fix.

Thursday night: a bill I did not see coming

Around $723, for databases.

That one was my own fault, and it is worth being straight about it. Every app we build can get its own database, and I do an enormous amount of testing, so I create an enormous number of them. I had cleanup scripts to tear them down afterwards. They were not running. Nobody told me, because nothing was watching them, and I was too busy to notice.

So I paid for months of test databases that nobody, including me, ever opened again.

Steps are in now so it cannot happen quietly again: the cleanup runs on a schedule on a machine that reports it, something counts the databases every day and says when the number moves, and the number is on a screen I actually look at.

I also signed a contract with Supabase for Platforms this week, which gives us a proper, supported way to create and manage those databases at scale rather than bolting it together ourselves.

Good timing, as it turned out.

Friday and Saturday: 3,496 databases

Completely separate problem, and a better story.

A bit of our code keeps a couple of spare databases warm so nobody waits when they hit build. It asks how many spares exist, and tops up if there are fewer than two.

The asking part had gone blind. Not broken, blind. Every time it asked, it got back a perfectly confident zero.

So it made two. Waited twenty seconds. Asked again, got zero again, made two more. All day, and all night.

By the time we caught it there were 3,496 of them sitting there that nothing would ever read.

An endless loop of database creation that sat unused but cost real money.

And I do mean we. An agent found it and brought it to me with the measurements already done. Left to me, it would have turned up on a bill again.

Here is what gets me. Nothing misbehaved. Every piece did exactly what it was told, the target of two was right, and even the log line looked healthy. The sentence nobody had ever written is the one that would have stopped it dead.

I have made three thousand of these and I cannot see a single one.

So I spent Saturday teaching the fleet defensive programming. Not finding something now means "I do not know" rather than "there is nothing", and nothing creates or deletes on "I do not know". If you make something, you have to see it afterwards or stop. Everything that spends, creates or deletes gets a hard limit.

The one that matters most to me is simpler than all of that. The worst thing we could ever do to somebody is lose their work. So anything touching a database or a customer's code now gets a second pair of eyes, and has to say in writing what the damage would be if every assumption in it turned out to be wrong.

We have to assume that anything that can go wrong eventually will. So we build defensive programming techniques into the system to catch it.

The good stuff

It was not all fires.

Vorx AI Ltd is a real company. The paperwork came through this week. Next job is the bank account, which is somehow the least exciting sentence I have written all month and I am still delighted about it.

Introducing "Build Engine 002".

I know, I know. A very sexy name, eh.

Here is where it came from. I got Fable to read back over the last hundred apps we have built and answer one question: how do we build these faster, and where are we doing the same work over and over?

The answer was uncomfortable and useful. An enormous amount of what an app needs is the same every single time. A list. A detail page. A form. A login. A table that sorts. We were paying a model to hand write every one of those from scratch on every build, and then paying again when it got one of them slightly wrong.

So 002 splits the job in two. The model writes a spec: what this app is, what it holds, what the screens are. Then an assembler builds it. The model still decides what the app should be. It just no longer hand writes every line of what ships.

We stopped asking a model to write the whole app and started asking it to decide what the app should be.

And here is the part I am genuinely excited about.

The expensive bit of a build was never the building. It is the repair rounds afterwards. The app comes out not quite right, so we go back around and fix it, and again, and again. That is where most of the money went and nearly all of the waiting.

Assembled parts do not need repair rounds. Across our comparison packs, every single repair round landed in model written code. Not one was in anything the assembler produced. Shrink the hand written surface and you shrink the reason to go back around at all.

Cost is not the story, by the way, and I will retire that argument myself before anybody else does: a build is about 25 cents on either engine, and in one head to head the old one came out dearer. The win is that it does the same thing twice and does not need fixing.

What it has already done, all measured, all blind or independently judged:

• It cleared our design floor outright, zero contrast failures and nothing outside tolerance, twice in seven comparison cycles.
• It won a landing page in a blind comparison for the first time, on both desktop and phone.
• Asked to show who is on which pitch, 002 built the calendar view. 001 built no calendar page at all.
• On those same comparison briefs, in the one pack we had judged, the old engine failed contrast 16 of 20, showed raw ISO dates on the one list an owner actually reads, and buried the number the owner most wants.

Where 001 still wins: the landing headline, narrowly, because its headline is a sentence and its stats fit on one line. I would rather say that than pretend the gap is closed.

I am not going to hand you a percentage, because I do not have an honest one yet. But the results are looking great and the decision is made, and it cleared its final gate tonight. It is not live yet and I am not going to pretend otherwise, so you will hear when it is. I am optimistic about this one.

And the fix for the school car park incident is the template for everything now. Do not write a note asking people to be careful. Build something that refuses.

I can see every penny now, and it was worth the engineering

This one deserves more than a line, because I think it is the most valuable thing we have built all month and it is completely invisible from the outside.

Every single AI call we make now lands in one database. Every call, everywhere. The fleet building Vorx itself, and Vorx building a customer's app. Whatever the model, whatever the provider, whichever machine it ran on. Cost, tokens, which agent, which project, which stage of which build, and what it was for.

That took real engineering effort and it was absolutely worth it, because for months I was running a margin business on numbers I could not see. I found out what things cost the way everybody finds out what things cost: on a bill, weeks late, when it is already spent.

For months I was running a margin business on numbers I could not see.

One day of the fleet: 8,816 model calls, priced, and sliceable by agent, by role, by machine, by provider, by model. This screen did not exist a fortnight ago.

This week alone it earned its keep. It is how I caught the repair step escalating to the most expensive model we have and quietly running over a hundred and fifty dollars a day, which is the kind of thing you can only act on if you can actually see it.

And it compounds. Every question I could not answer last month is now a query. Which model is worth its price on which job. What a build actually costs us against what we charge. Which stage of the pipeline eats the budget. Where the margin goes. In the long run this pays for itself over and over, because every single decision I make from here about models, pricing and where to spend gets made on real numbers instead of on a feeling.

Running a software company turns out to be mostly cost control. Then again, it always was.

Some design notes

I have not been obsessing over every pixel these last two weeks the way I usually do. I still could not leave it alone entirely.

The templates were too hidden. They are genuinely excellent and they were buried in the Launchpad, so I moved them into the main navigation. That is an interesting balance to strike. I am trying to keep this app elegant and simple, and every single icon in the main menu has to fight hard to earn its place. They earned it. They also got a redesign in the app and a few tweaks while I was there.

The templates page after the move and the redesign. Fifty six of them now, and one click from anywhere instead of buried.

Design System moved out of Settings and onto its own page. It was a section inside the settings panel and it is far too important for that. It needs the breathing room. But I did not want it in the main menu either, because most companies do not need it on day one, so it lives in the profile dropdown. That is the kind of call there is no clean answer to.

Project Settings became a dialog. I went back and forth with myself on this one for weeks. Like most things in design there are real pros and cons on both sides. What settled it is that from the projects list you can now adjust a project's settings without going into the project at all, and I love that.

And I added more keyboard shortcuts and a command palette. Hit it from anywhere and search your apps, your pages and your settings.

Where the three big things actually are

Eighty test builds this week, every one of them green, and what each one cost me sitting right beside it. A month ago I could not have told you either number.

This week we go to battle against updates

This is the one that keeps me up.

An update is when you change an app you already have. A build can only disappoint you once. An update has to understand everything already there and then not break any of it. That is where people actually live, and it is our weakest thing by a distance.

So I stopped asking our own pipeline how it was doing and went and looked.

The middle one is the one that stopped me. Of those twenty one thumbs, exactly ONE could be set against anything our own checker had said. Fourteen came before we were recording the run at all, and on six more the checker simply never spoke. So it is not mainly that our instrument was wrong about her. It is that we asked her a question and, for almost all of it, our own instrument had said nothing whatsoever.

The third one is not a measurement problem. That is a person, pressing thumbs down nineteen times, on the record, telling us exactly where we were failing her. Nobody ever read that column.

She told us nineteen times, inside the product, and nothing was listening.

The plan, in this order. Fix the judge so it reads the actual change, file by file. Score it against her nineteen thumbs, which is a free calibration set we already owned. Answer every update honestly, because a refusal is a reply and nobody should be charged for work that was never checked. And make the checks actually run: about a sixth of updates were skipping them entirely. That last one is fixed as of today, and it took one login page from 6.7 seconds to half a second.

One more thing I will say because it is the most useful thing we learned. We went looking for our checker's mistakes and mostly did not find them there. Six of the problems were in what it was HANDED rather than in how it reads: in one case a run flipped to correct with no change to the checker at all, purely because the text it had been given opened with an unrelated server error that made a perfectly good edit look like it had broken the build. Before you blame the judge, look at what the judge was shown.

This week we go to battle against updates.

And the database move is done

Cost control again, which is most of this job now.

Our production database was sitting on a 628 GB disk holding about 1.8 GB. An automatic resize with no ceiling on it, growing roughly 20 GB a day for a while, and once a disk is that size you cannot shrink it. The only way back is a new machine and a careful move of the one database everything reads from. That is why it never happened by itself, and it is done as of tonight.

It was not clean. About ten minutes in, the board my own agents work from went down: it came up pointing at a database that was not there yet. No customer touched it, builds and sign in kept answering, but for about twenty five minutes not one of my agents could pick up a task. And fittingly for this week, the first thing every single one of them did was hand me a confident answer that was out of date rather than say it could not tell.

The bar, same as always

I want Vorx good enough to hand to friends and family without worrying about what happens next.

It was a rough week. But every single thing in it was something invisible becoming visible, and I would rather have a frustrating week where I find four of those than a calm one where I find none.

That is still the hard part. Not the building. The knowing.

And one more thing...