How We Evaluate Our AI: Scenarios, Benchmarks, and Continuous Quality Tracking

We're Biscuit, a platform that turns your idea into a real business. App, payments, hosting, support, all of it. To do this, we run many models. This post is about how we evaluate every model, tracked over time. We know which model wins on quality, which wins on speed, and where the best tradeoff lives. We used to ship and hope. Now we ship and know.

We don't run a single prompt. We run many different models across many different tasks: one for generating UI code, another for planning data schemas, another for debugging, another for feature suggestions. Each model has its own strengths, and each task has its own requirements. Every prompt tweak, every model swap, every new workflow could make one thing better and break another.

A few months ago, shipping a change looked like this. An engineer would update a prompt or swap a model. They'd test a few things by hand. It looked fine. They'd ship it. Then they'd hope. Hope it didn't break something unrelated. Hope the ten other things that model needed to do still worked. There was no system. Just skilled people doing their best with manual spot checks.

The problem is that AI failures are quiet. The AI doesn't crash. It doesn't throw an error. It just does half the job, or does the wrong thing confidently.

One day we asked the AI to add a priority field to a todo app and display it as colored badges. It added the field. But the badges didn't show up in the UI. The data was there, the code compiled, but the feature was half-built. You don't notice these until you happen to try the exact task that broke.

Our platform handles hundreds of different interactions across every kind of app. Editing code, clicking buttons, filling forms, creating data models, generating features. We needed a way to capture each of these as a repeatable, automated test. And we needed to run them constantly.

So we built one.

We turned the priority-field problem into what we call a scenario: a self-contained test case for the AI. A scenario sets up a specific context, gives the AI a prompt, and then verifies whether the AI actually did the whole job.

What makes a scenario useful is that it captures the full context the AI needs. Not just a prompt, but an entire mini-app to work with, a specific starting state, and a way to check the result. This means we can replay the exact same test, against any model, at any time.

Here's what one looks like in practice:

Some scenarios have clear-cut success criteria. Does the code compile? Is the priority field in the data model? Does the view contain an element with the right CSS color? For these, we write deterministic checks that inspect the actual code, data store, or rendered output. Pass or fail, no ambiguity.

But many of the things our AI does are harder to judge with a simple assertion. Did it write clean code? Are the feature descriptions concise? Did it find the root cause of a bug, or just describe a symptom? For these qualitative outcomes, we use LLM-as-a-judge: a separate model evaluates the result against a structured rubric.

LLM-as-a-judge is exactly what it sounds like: you use a strong language model to evaluate the output of another model. Early work in this area, including MT-Bench and Chatbot Arena[1] and G-Eval[2], showed that the approach can correlate well with human assessments, but also surfaced real failure modes: judges can be biased toward verbose answers, toward their own outputs, or toward whichever option appears first. The field has matured quickly since then. For a good overview, Li et al.[3] offer a comprehensive survey of where the field stands.

The key lesson from this research is that how you ask the judge matters enormously. A vague "rate this from 1 to 5" prompt produces noisy, inconsistent scores. What works is giving the judge a detailed rubric with explicit criteria, concrete score-level descriptions, and decomposed dimensions. This is well-supported: LLM-Rubric[4] showed that multidimensional rubrics significantly improve agreement with human ratings. "FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets."[5] decomposed evaluation into fine-grained skills like readability and conciseness. CheckEval[6] found that binary checklist questions reduce variance and improve interpretability. And LMUNIT[7] went further by turning criteria into natural-language unit tests, an idea that maps naturally to the kind of technical and style constraints we care about.

Our judge follows the rubric-based approach. Each rubric defines weighted criteria on a 1–5 scale, and every criterion has explicit descriptions for every score level. Instead of interpreting vague labels, the judge matches the output against concrete descriptions of what a 3 versus a 5 looks like for that specific criterion. Individual criteria can also enforce minimum thresholds, so a scenario can require at least a 4 on correctness even if style scores are more lenient. This is especially powerful for evaluating feature generation quality, code style, and complex editor tasks where "correct" is a spectrum.

This approach isn't perfect, and we're careful about where we trust it. Large-scale studies[8] show that judge reliability varies significantly across task types and the properties being evaluated. Context-grounded judging[9], such as checking whether generated code actually follows a codebase's conventions, is substantially harder than generic quality assessment. We account for this by tailoring criteria to each scenario type rather than relying on a single universal rubric.

There's much more to say about designing effective LLM judges, from rubric calibration and bias mitigation to choosing judge models, ensemble approaches[14][15][8], and prompt engineering for evaluators. We'll cover those in detail in a follow-up post. For now, the takeaway: structured, criterion-level rubrics turn "does this feel right?" into something repeatable and measurable.

One scenario tells you whether one task works. That's useful, but it doesn't answer the bigger questions. Is the AI good at forms? Is it getting better at code editing? How does it compare across all UI interactions?

A benchmark answers these questions. It's a curated collection of scenarios, organized by capabilities.

When a benchmark runs, you don't just see a single pass rate. You see a breakdown by capability, and this immediately tells you where to focus. A single number hides too much. Capabilities make the score actionable.

A capability groups scenarios that test the same skill, so you can track it independently across models and over time. If the score drops, you know exactly where to look. We try to make capabilities meaningfully distinct from each other: things like "handles multi-file changes" or "follows design tokens," not internal implementation details that users never see. Getting the granularity right takes a few tries: too broad and problems get averaged away, too narrow and you don't have enough scenarios to say anything useful.

We organize benchmarks into three tiers. Smoke benchmarks are 12–15 scenarios that run in minutes on every PR, serving as a quick sanity check. Core benchmarks cover essential capabilities like UI basics, form handling, code editing, and bug fixing. Extended benchmarks provide comprehensive coverage: multi-turn conversations, long context, multilingual support, complex tasks. Smoke runs on every change; core and extended run nightly.

Between scheduled runs, we also trigger benchmarks based on what changed. If files matching certain patterns are modified, the relevant benchmarks run automatically. This means we don't have to wait for the next nightly run to catch a regression: we get a signal within minutes of the change landing.

After every benchmark run, the system automatically compares against the previous run on the same benchmark, model, and branch. Each scenario is classified: regressed, improved, or stable.

But here's the subtle part: with AI, some amount of variation is normal. A model might fail a scenario one run and pass it the next, purely due to sampling randomness. So how do we tell real regressions from noise?

Early on, we hit a frustrating problem. A benchmark score would drop from 92% to 78%, we'd spend an hour investigating, and it would turn out someone had changed the test, not that the AI got worse. The trend line was untrustworthy because we couldn't distinguish "the AI got worse" from "we changed the test."

So every scenario and benchmark gets a content hash (SHA-256 of the resolved definition). When a scenario's prompt or verification logic changes, the hash changes. This lets us distinguish "the AI got worse" from "we changed the test." Without this, trend lines would be untrustworthy.

Before any statistical comparison, the system checks that the content hashes match and that no scenarios were added or removed. If they don't match, the comparison is marked not_comparable, so no misleading p-values and no false alarms.

To make this concrete, let's look at one benchmark in detail: the editor benchmark. It tests the agent end-to-end: give it a task, let it work, then check whether the result is correct. It covers dozens of capabilities across the everyday work the editor performs, from fixing bugs and building UI components to managing state and multi-turn conversations.

To give you a feel for what these scenarios look like, here are a few examples:

A pass rate of 80% on 10 scenarios means something very different from 80% on 100 scenarios. With 10 scenarios, the true capability of the model could plausibly be anywhere from 50% to 95%. With 100 scenarios, it's much tighter: probably between 71% and 87%.

We use Wilson score intervals to put uncertainty bands around every pass rate. Try it yourself. Drag the sliders to see how sample size and pass count affect confidence:

Today, when we notice the AI half-building a feature, here's what happens. We write a scenario. It joins a benchmark automatically through its tags. The next CI run picks it up. If the fix works, it shows as an improvement in the comparison view. If it regresses later, Slack tells us within minutes.

We ship prompt changes and model updates with confidence because we know within minutes whether something broke. We choose between models based on data, not intuition. We can see at a glance whether our AI is getting better at forms, faster at code editing, or worse at multi-turn conversations.

We can tell you how every model scores today. We can't yet tell you how they'll score next quarter.

The building blocks are there. The next step is to curate a set of "frontier scenarios". Real tasks we actually want solved, where today's best models succeed maybe 30% of the time.

Once you have enough of those, you can start asking a more interesting question: how fast is the frontier moving? Run every new model release against the same battery. Track what fraction of frontier scenarios cross the "50% reliability threshold" over time. Fit a curve. This is the approach behind work like METR's autonomy evaluations: build a hard-enough benchmark, measure it consistently, and let the trendline tell you what's coming.

If we can predict that a capability (say, multi-file refactoring) will be reliably solvable in six months, we can start building for it now. Ship the feature just as the model catches up.

We're not there yet. But the evaluation system described in this post is the foundation it would run on.

We're honest about the limitations. We don't yet run the same scenario multiple times to estimate non-determinism; each scenario runs once per benchmark execution, so we can't separately measure "is this failure real or just sampling noise." We don't apply multiple testing correction across trend comparisons, which means if you look at enough charts, you'll find a "significant" result that isn't. And capability-level pass rates on small groups (2–5 scenarios) have confidence intervals so wide they're almost decorative.

These are known trade-offs. The system is designed to give us useful signal at realistic scale, not to be a peer-reviewed statistical study.

We used to ship and hope. Now we ship and know.