Working with AI. Building and checking work I didn’t write myself.

The AI Finished. I Still Had to Check the Work.

AI helped me build infrastructure tools, web apps, and an iOS app. It also produced work that looked finished and wasn’t. I had to learn how to check it as carefully as I directed it.

By Scott Davis. Based on the recorded project account; wording updated September 6, 2026.

How did I know the result was right?

Executive perspective

Why does this matter?

I could take on work I couldn’t have done alone before.

With AI, I built infrastructure automation, community systems, web applications, and a mobile app. Some of that was well outside my usual work. I could learn the unfamiliar parts as I went instead of needing to know every tool before I started.

More work arriving meant more work to check.

I had to explain the need, give the AI enough background, and split the work into jobs with a clear beginning and end. I also had to decide how to check each result. On the cemetery repairs, eleven AI workers ran in parallel with executed checks. Production-only failures still turned up afterward.

Sometimes I’d built too much.

I built an assistant-memory system with a database, a model router, a chat interface, an MCP connection, and a mobile front end. Later I replaced it with plain markdown files. The part I really needed to do was decide what was worth remembering and write it clearly enough to use later.

Architecture perspective

Why was it designed this way?

I had to explain the job before handing it off.

I wrote down the relevant environment, what the task could change, and what would prove it worked. Each worker needed enough context to do useful work without guessing. Splitting up the jobs helped me do more at once, but only if they still added up to what I wanted.

Two workers couldn’t own the same files.

In the recorded workflow, each worker had a separate set of files. Failed checks supplied output for the next attempt, and I inspected some passing work by hand. I decided how to divide the work and whether the combined result was finished.

I could remove things I’d already built.

The memory system worked, but useful written context didn’t need all those layers. Once I understood that, I moved to a smaller setup. Being able to build something didn’t mean I needed to keep running it.

Technical perspective

How does it actually work?

The instructions included the checks.

Each task specified the work, the relevant environment, the files it owned, and commands to verify the result. A failed check fed its output into the retry. I compared content rather than incidental formatting and checked production telemetry after deployment. AI helped write and inspect the code; I still had to read the results.

These were observations from my own work.

The detailed account includes a check that rejected correct page content, model results from a particular set of runs, and production bugs that passed local checks. Those examples changed how I worked. They don’t establish how every model or test will behave.

The detailed project account

These details come from the account written before the September 2026 site update. I’ve rewritten the wording, but kept the measurements, findings, and qualifications. “Today” and other time references describe that earlier period. I haven’t remeasured the results or assumed that open issues were fixed.

What was needed

AI helped me build the projects on this site faster than I could have done alone. That included infrastructure automation, monitoring, encampment apps, and cemetery records. On the cemetery system, I audited and repaired the production application in two days.

That also left me with a lot of work to check.

AI could produce a migration, a test suite, or a thousand lines of a health check faster than I could read them. Its completion message didn’t reliably tell me whether the result was correct. I’d seen wrong work reported with the same confidence as right work.

I needed a repeatable way to check the work as it arrived. Otherwise I was just collecting code I hadn’t had time to understand.

I’d seen similar problems elsewhere: a storage check defaulting missing data to OK, a score reporting 100 percent while seven of fourteen vCenters were unreachable, and a web handler returning HTTP 200 while emails failed. An AI worker saying it was finished could give me the same false reassurance. Across those four systems and two languages, I needed to check what had actually happened.

How I work

I used a tool I didn’t build to assign work to AI workers and run checks. I supplied the task instructions and decided what the checks needed to prove.

For the runs described here, I used these rules:

Each task had an executed check command. A zero exit code was required to pass. I didn’t accept the model’s summary or a plausible-looking diff as a substitute for running it.

The check needed to examine the work. test -f report.md only proved a file existed. Depending on the task, I ran code or tests, searched the output for required content, or compared it with the source.

A failed check had to print the reason. That output went into the next attempt, so the worker had something useful to correct.

Each worker owned named files. With eleven workers in the same repository, I assigned separate file sets in the manifest so two workers wouldn’t overwrite each other.

I read failures before retrying. If a task passed on the second attempt, I looked at whether my first instructions had been unclear. I couldn’t assume the model was the only problem.

I also inspected at least one passing task by hand each run. Passing the command was required, but the command could miss something.

A lot of the work was writing context. These workers didn’t retain prior sessions and couldn’t ask questions, so their instructions had to stand alone. I kept a written description of the environment and sanitized reference material so they had accurate information without private details they didn’t need.

What it produced

For the cemetery records system, eleven workers helped me audit and repair it in two days. That included nine guarded database migrations, rebuilt permissions, a nineteen-test integration suite, and checks against production. Each task was checked before merging.

The original version of this site had five pages. A separate worker built each page and owned one file; a sixth built the design system. I compared the rendered writing with my source paragraph by paragraph and number by number. A summarized sentence, dropped paragraph, or missing figure failed the task. That check was meant to keep the original writing intact during page construction.

I also built a functioning financial management app for iOS. I’m an infrastructure engineer, and mobile development wasn’t my background. AI let me work through the unfamiliar tools while building the application. That made a different kind of project possible for me.

For assistant memory, I built a model router and hosted database, used a chat workspace as the interface, connected them through an MCP server, and added an iOS front end. The idea wasn’t mine, but I built the system. It worked, and I later moved off it.

I replaced it with a directory of plain markdown files.

I’d put a lot of work into storage, retrieval, and interfaces. What I really needed was to decide what was worth remembering and write it clearly enough to be useful six months later. Once I understood that, I didn’t need all those layers.

Where it goes wrong

Three failures changed how I worked:

All eleven workers passed their checks on the cemetery repairs. The test suite passed and deployment succeeded. Production still had two bugs: the hand-edited lockfile caused MODULE_NOT_FOUND on every email while the handler returned HTTP 200, and an authentication assignment worked locally but silently did nothing on the deployed worker. My code review and local checks hadn’t caught them. Querying production telemetry after deployment did.

On the original site build, one page failed twice even though its text was correct. Extracting text around an emphasis tag added whitespace, changing absolutes, which to absolutes , which. My check was rejecting punctuation spacing. I changed it to compare word sequences so it still caught dropped or rewritten text without rejecting harmless formatting.

I logged each check result by model and task type. In that run history, one inexpensive model passed code fixes and code reviews on the first try every time, but passed page-building tasks on the first try only fifty percent of the time. That wasn’t what I expected. I used the measured first-attempt results to assign work instead of relying on reputation. Those observations describe those runs, not a guarantee about the model.

What I learned

I required the check to run and exit zero before accepting a task. The worker saying it was finished wasn’t enough, and I still had to understand what the check proved.
I spent time making the check capable of rejecting wrong work. Clear instructions helped, but a well-written request couldn’t tell me whether the result was correct.
I made content checks strict about missing or changed meaning and tolerant of harmless formatting. A check that repeatedly rejected good work would be hard to trust.
When a retry passed, I reviewed the instructions as well as the code. With workers that couldn’t ask questions, something I’d left unclear could easily cause the first failure.
Eleven workers let me handle separate tasks at once. That helped because each had a defined job and a check. I still had to review how the pieces worked together.
The memory project reminded me to ask what I actually needed. A database, chat interface, MCP server, and mobile app hadn’t solved the job of deciding what to remember. I still had to write that down clearly, even after moving to markdown.
I needed to decide what finished meant before accepting the result. An empty result set could be a problem. HTTP 200 could hide a failed action. No error message didn’t necessarily mean success. AI helped me build the work; I still had to recognize those cases and investigate.