What we learned putting a model in front of failed build logs

ZevAI explains why a deploy failed. The useful part was not the prompt. It was deciding when not to call the model at all, what leaves your network, and how you find out whether any of it helped.

We shipped ZevAI a few weeks ago. Its first job is explaining failed deployments: press a button, get back what happened, why, and what to do next, grounded in our own documentation.

That post is what it does. This one is how it works, and it is written for anyone putting a model in front of machine output of their own.

The interesting part was not the prompt. Prompts are the easy bit. The interesting part was a set of decisions about when to call a model, what to send it, and how to know afterwards whether it helped anyone.

Deterministic first, model for the long tail

Before any of this existed we had a regex analyzer over build logs. Perhaps thirty patterns, each matching a failure we had seen often enough to write down: a missing lockfile, a Node version mismatch, a port the app never listened on.

The temptation when you add a model is to replace all of that. We kept it, and it is the first thing that runs.

Patterns are free, instant, and cannot be wrong in a creative way. When one matches, the customer gets a precise answer with no inference involved. The model exists for everything else, which in practice is most failures, because the long tail of ways a build can break is genuinely long.

It also turns out the analyzer makes the model better. When a pattern does match, we pass what it found into the prompt as evidence rather than letting the model rediscover it. Cheap context beats clever prompting.

Never let it answer when you already know the answer is "us"

This one cost us nothing to implement and would have cost a lot to skip.

Some failures are ours. A build host that refused a connection, a registry we could not reach, an internal timeout. The log looks alarming, and it has nothing to do with the customer's code.

Hand that log to a model and ask why the build failed, and it will find something in the customer's repository to blame. It is doing what it was asked: explain this failure. It does not know that the interesting line is a symptom of our infrastructure having a bad minute, and it will confidently suggest the customer change a dependency that was never the problem.

So the explainer checks first. If the failure is platform-side, it answers from text we wrote, says plainly that this one is on us, and never calls the model at all. Same rule for failures where the message in the log is one our own system wrote.

The general form of the rule: if you already know the answer, do not ask. A model asked to explain something it cannot see the cause of will produce an explanation anyway, and a confident wrong answer about whose fault something is costs more trust than no answer.

Decide what leaves your network

Logs are not clean. They contain environment variable names, sometimes values, internal hostnames, the names of things running on the machine that are none of the customer's business and certainly none of a third-party API's.

Three steps, in order, before anything goes out:

Sanitise. Strip infrastructure detail. The same pass the dashboard already uses before showing logs to customers.
Redact. Anything shaped like a credential goes, before the log crosses any network boundary.
Clamp. Head and tail only. A model does not need the middle two thousand lines of a dependency install, and sending them costs money and makes the answer worse.

Worth being precise about the standard we set: the log that leaves our network is the log the customer can already see, minus anything that looks like a secret. That is a sentence you can hold yourself to, and it decides most of the individual questions for you.

Ground it, then check its work

An ungrounded model writing about your product invents your product. It will suggest configuration options you do not have, and file paths that do not exist.

So the explainer retrieves from our own documentation, matched to the failure, and the prompt is built around those pages. The answer cites them, and the dashboard links them, which means a wrong answer is at least a checkable one.

Two details that turned out to matter more than expected:

Track whether grounding actually happened. If the documentation corpus is unavailable, the answer comes from the log alone. That is still useful, but it is a different thing, and the response carries a flag saying so rather than pretending.

The prompt forbidding something is not a guarantee. We do not want generated text naming infrastructures. The prompt says so. We also scrub the output afterwards against a brand list, because a prompt is a request and a scrub is a rule. Everything the model produces passes through it before any customer sees it.

Cache the obvious thing

A failed deployment's log never changes. It is the most cacheable object in the product, and the first explanation is the only one that has to be generated.

We cache for 24 hours, keyed to the deployment. Reopening the page is free, which matters because people reopen the page constantly while they are fixing something.

You cannot tell if it works unless you record it

Every call is written down: which deployment, whether it was cached, which patterns matched, what it cost, whether the answer was grounded. The card has thumbs up and thumbs down, and that rating goes on the same record.

That gives us an adoption number, a helped-rate, a scrub rate, and a cost per explanation on one admin page. Without it we would be guessing about a feature that costs real money per use, and "it feels useful" is not a number you can act on.

The scrub rate is the one I did not expect to care about. If it climbs, the prompt is drifting, and I would rather find that in a chart than in a screenshot from a customer.

Leave a door out

The failure mode of any support-adjacent AI feature is becoming a wall between the customer and a human.

When the explainer says the failure was ours, the card leads with Report to ZevCloud. One press attaches the deployment, the log excerpt and the verdict to an alert our team sees, and hands the customer a reference number. When it was a project-side failure the same option is there, just quieter.

The AI is a first answer, not a gate. If it is wrong, or right in a way that does not help, getting to us should take one click and no explaining.