Proof Before Ship: How Skill Evals Turn AI Agents from Guesswork into Engineering

Part 1

The Discipline Gap: Why "It Looks Good" Is Not Enough

Software engineering solved this problem decades ago. Before CI/CD, developers shipped code based on local testing and manual review. It worked until it did not. The industry converged on automated test suites, quality gates, and reproducible builds — not because engineers lacked judgment, but because judgment alone does not scale.

AI skill authoring is where software was before continuous integration. Most teams write a SKILL.md file, try it a few times, and ship it if the outputs "look right." Forrester's 2026 testing report put it bluntly: organizations need to "test their AI agents — like, at all." The ones that adopt AI-infused testing tools report measurably higher automation coverage and fewer production defects.

The numbers tell the story. McKinsey's 2026 State of AI report found that 92% of organizations plan to increase AI investment. But only 40% have scaled beyond pilots. The gap between investment and value is not a model problem. It is a verification problem.

The eval gap in practice

Galileo AI's 2026 evaluation engineering report identified the critical issue: performance on pre-deployment tests does not reliably predict real-world utility. The "evaluation gap" between what works in a notebook and what works in production is the primary risk factor for AI deployments — and it widens when teams skip systematic evaluation entirely.

The discipline that closes this gap has a name. Practitioners call it eval-driven development (EDD) — a direct analogy to test-driven development. The core principle: prove that your skill performs inline with current metrics or better, before you accept it as a solution.

Part 2

Anatomy of a Skill Eval: The Skill-Creator Framework

Anthropic's skill-creator plugin for Claude Code provides the most complete reference implementation of eval-driven skill development. Released in March 2026, it codifies a five-phase loop that treats every skill change as a hypothesis to be tested.

Figure 1. The skill eval lifecycle as a negative feedback loop. When pass_rate falls below threshold, the system feeds failure signals back into skill improvement — teams improve prompts and gates, not blame models.

The loop works like this:

Draft the skill. Write or modify a SKILL.md file with structured instructions.
Run test cases. Spawn parallel subagents — one with the skill, one without (or with the previous version). Each test case gets both configurations so you always have a baseline.
Grade outputs. A grader agent evaluates each assertion against the outputs. Code-based graders handle deterministic checks; LLM-as-Judge handles semantic ones.
Review results. An interactive viewer presents qualitative outputs and quantitative benchmarks side by side. The human reviews, leaves feedback, and identifies what needs to change.
Improve the skill. Apply changes based on evidence, not intuition. Rerun. Repeat until the evidence says yes.

The Negative Feedback Loop: A Paradigm Shift

This is not just a workflow diagram. It is a negative feedback loop in the control theory sense — and recognizing it as such changes how your team thinks about AI quality.

In a negative feedback loop, the system's output is measured against a target, and the error signal (the gap between actual and desired performance) is fed back to adjust the input. The critical shift: when a skill underperforms, the fix is not to switch models or solve the problem manually. The fix is to improve the prompt, tighten the quality gates, and raise the artifact quality.

This inverts the typical failure pattern. Most teams hit a bad output and react in one of two unproductive ways: they blame the model ("Claude can't do this") or they bypass the skill and solve the problem by hand. Both responses break the feedback loop. The eval-driven approach keeps the loop closed — every failure becomes a signal that improves the next iteration of the skill itself.

What the feedback loop actually improves

Each iteration through the loop refines three things simultaneously: (1) the skill prompt — clearer instructions, better examples, more precise constraints; (2) the quality gates — sharper assertions, better-calibrated thresholds, more discriminating tests; and (3) the artifact quality — the evals.json, grading rubrics, and benchmark configurations that make the entire process reproducible. Over time, the system converges on reliably high performance — not through heroic individual effort, but through systematic error correction.

The baseline is the key

Every eval run compares two configurations: with_skill and without_skill (or old_skill for improvements). This is not optional. Without a baseline, you cannot distinguish "the skill helped" from "the model was already good at this." The delta between configurations is the actual measurement of skill value.

This Is Not New: PDCA, DORA, and Managing Delivery as Process

If this loop looks familiar, it should. It is a PDCA cycle — Deming's Plan-Do-Check-Act, the foundation of every process improvement methodology for the past 70 years. The mapping is direct:

Plan — Define expectations and acceptance criteria (evals.json)
Do — Run the skill against test cases (with_skill vs without_skill)
Check — Grade outputs, analyze benchmark delta (grading.json, benchmark.json)
Act — Improve the skill prompt, tighten gates, standardize what works (SKILL.md updated)

The best engineering teams already manage software delivery this way. Google's DORA research (DevOps Research and Assessment) proved that elite teams differentiate themselves on four metrics: deployment frequency, lead time for changes, change failure rate, and mean time to recovery. The 2024 DORA report found that AI adoption improves deployment frequency — but can increase instability without strong process foundations. The eval loop provides exactly that foundation for AI skill delivery.

Six Sigma's DMAIC (Define-Measure-Analyze-Improve-Control) maps just as cleanly. The Measure phase is your baseline. The Analyze phase is your analyst pass flagging non-discriminating assertions. The Control phase is your quality gate with an acceptance threshold. These are not analogies — they are the same discipline applied to a new artifact type.

The shift: AI delivery is process management

The teams that treat AI skill authoring as a process to be managed — with metrics, feedback loops, and continuous improvement — will outperform the teams that treat it as a creative act. This is the same transition software engineering made from "artisan coding" to CI/CD pipelines. The Toyota Production System, Kaizen, PDCA, DORA — these frameworks exist because the discipline of systematic improvement always outperforms the heroics of talented individuals. Eval-driven development is how that discipline arrives in AI.

SkillsBench, the first comprehensive skill benchmark (86 tasks, 7 models), quantified the impact: structured skills improve pass rate by +16.2 percentage points on average. But the variance is enormous — from +4.5pp in software engineering tasks to +51.9pp in healthcare. This means you cannot assume a skill works. You have to measure it in your domain.

Part 3

The Data Structures That Make Evals Reproducible

Eval-driven development requires reproducible evidence. The skill-creator framework achieves this through five JSON schemas that compose into a complete audit trail. Here is what each one captures and why it matters.

evals.json — The Test Suite

Defines the prompts, expected outputs, and verifiable expectations for each test case. This is your test harness.

                JSON
                evals/evals.json
            

{
  "skill_name": "deploy-to-cloud",
  "evals": [
    {
      "id": 1,
      "prompt": "Deploy this Express app to Cloud Run with autoscaling",
      "expected_output": "Dockerfile + cloudbuild.yaml + working deployment",
      "expectations": [
        "Output includes a multi-stage Dockerfile",
        "cloudbuild.yaml has the correct project ID",
        "Autoscaling is configured with min 1, max 10"
      ]
    }
  ]
}

grading.json — The Evidence

The grader produces a per-expectation verdict with evidence. Each assertion is either passed or failed, with a quote from the output proving the judgment. This is the chain of evidence.

                JSON
                eval-0/with_skill/grading.json
            

{
  "expectations": [
    {
      "text": "Output includes a multi-stage Dockerfile",
      "passed": true,
      "evidence": "Found 'FROM node:20-slim AS builder' and 'FROM node:20-slim AS runtime' in Dockerfile"
    }
  ],
  "summary": { "passed": 3, "failed": 0, "total": 3, "pass_rate": 1.0 }
}

benchmark.json — The Statistical Summary

Aggregates results across configurations and runs into mean ± stddev statistics. This is where you see the actual skill impact.

Benchmark anatomy: pass rate, execution time, and token usage for with_skill vs without_skill configurations

Figure 2. The three dimensions of a benchmark comparison: pass rate (quality), execution time (cost), and token usage (efficiency). The delta between with_skill and without_skill is the measured value of the skill.

The benchmark includes a run_summary with mean and standard deviation for each metric. Multiple runs per configuration (typically 3) catch variance. The delta section shows the net impact: +50 percentage points in pass rate, at a cost of +13 seconds execution time and +1,700 tokens. That trade-off is the decision point.

comparison.json — The Blind Judge

For rigorous A/B comparison, a blind comparator agent scores two outputs without knowing which used the skill. It produces rubric scores across content (correctness, completeness, accuracy) and structure (organization, formatting, usability) dimensions. The winner is chosen on evidence, not labels.

history.json — The Iteration Record

Tracks the full version progression: which version beat which, the pass_rate at each step, and which version is the current best. This is the commit log of skill improvement.

Why JSON schemas matter

These are not just logging artifacts. They are machine-readable evidence that composes into an audit trail. When the EU AI Act enforcement date arrives in August 2026, organizations deploying high-risk AI systems will need documented quality management systems. Structured eval artifacts provide that documentation.

Part 4

Three Evaluation Approaches Compared

The skill-creator framework combines three distinct evaluation methodologies. Each has different strengths, and the right approach depends on what you are measuring.

	Code-Based Graders	LLM-as-Judge	Blind A/B Comparison
Deterministic	Yes	No (variance)	No (variance)
Semantic coverage	Narrow	Broad	Broad
Cost per eval	Negligible	1 LLM call	2 LLM calls + comparator
Eliminates bias	N/A	No (position bias)	Yes (blind labels)
Best for	Format checks, compilation, field presence	Code quality, document structure, explanations	Proving version A beats version B

1. Code-Based Graders

Best for: Assertions that can be checked programmatically. Does the output contain a file with a specific format? Does the code compile? Does the API response include required fields?

The skill-creator explicitly recommends writing scripts for programmatic assertions: "scripts are faster, more reliable, and can be reused across iterations." This is the evaluation equivalent of unit tests — fast, deterministic, and cheap.

2. LLM-as-Judge

Best for: Semantic evaluation where the "right answer" is not a simple string match. Did the skill produce a well-organized document? Is the code idiomatic? Does the explanation make sense?

Recent research has converged on best practices. A 2026 study found that a 0-5 grading scale yields the highest agreement between LLM judges and human evaluators (measured by ICC). The Judge Reliability Harness, an open-source framework, provides validation suites to stress-test judge reliability across agentic and free-response scenarios. The key finding: providing both reference answers and score descriptions is critical for reliable evaluation.

"Evals skills for coding agents guard against common mistakes I've seen helping 50+ companies."

— Hamel Husain, who has helped 50+ companies and trained thousands of engineers in eval-driven development

3. Blind A/B Comparison

Best for: Deciding whether a new version is actually better. The comparator agent receives two outputs labeled "A" and "B" without knowing which used the skill. It judges quality on a rubric, produces per-dimension scores, and declares a winner with reasoning.

This eliminates position bias and label bias. The analyzer agent then reads the transcripts to identify why the winner won — producing actionable improvement suggestions with priority and expected impact.

The hybrid approach

The skill-creator does not force a choice. It composes all three: code graders for deterministic checks, LLM-as-Judge for semantic evaluation, and blind A/B comparison for rigorous version comparison. Start with code graders. Add LLM-as-Judge when you need semantic coverage. Use blind A/B when you need to prove one version beats another.

Part 5

From Test Results to Proof: The Quality Gate Pattern

Running evals is necessary. It is not sufficient. The discipline lies in what you do with the results: establishing acceptance criteria before you run, and holding yourself to them after.

The "proof before ship" workflow has four steps:

Establish a baseline. Run the current version (or no skill) against your test suite. Record pass_rate, time, and tokens. This is version v0.
Make the change. Edit the skill. This is your hypothesis: "this change will improve pass_rate without degrading time."
Re-evaluate. Run the modified skill against the same test suite with the same configuration. Record the delta.
Accept or reject. If pass_rate meets the acceptance threshold and the delta is positive, accept. If not, iterate or reject.

Skill iteration progression showing pass rate improvement from 35% at baseline to 87% at version 3, crossing the 80% acceptance threshold

Figure 4. Iteration progression of a hypothetical skill improvement. The baseline (v0, no skill) starts at 35% pass rate. Each iteration improves the skill until v3 crosses the 80% acceptance threshold. Note the modest time increase — better skills use more tokens but produce higher-quality outputs.

Statistical Rigor: Multiple Runs Matter

LLM outputs are non-deterministic. A single run tells you what happened once, not what typically happens. The benchmark schema includes a runs_per_configuration field (typically 3) specifically for this reason. Mean ± stddev catches variance that a single pass/fail hides.

Catching Non-Discriminating Assertions

The analyst pass after grading specifically looks for assertions that pass 100% in both configurations. An assertion like "Output is a valid JSON file" might always pass regardless of the skill — it tests the model's baseline capability, not the skill's contribution. These assertions are noise. The analyst flags them so you can replace them with assertions that actually discriminate skill value.

The Enterprise Quality Gate

BCG's RATE.ai framework validates this pattern at enterprise scale. Projects that pass their structured gate — with legal, risk, and IT sign-off built into the process — reach production 30% faster than those that try to skip ahead. The gate is not a bottleneck. It is an accelerator. And with the EU AI Act enforcement date approaching in August 2026, documented quality management systems become a regulatory requirement for high-risk AI deployments.

The cost of not evaluating

Enterprise AI development is expensive and high-stakes. Production deployments without structured evaluation routinely face significant cost overruns. Deloitte found that the vast majority of companies are still failing at AI production deployment — and the winners differ from the losers in one specific capability: systematic evaluation before deployment.

Part 6

The Emerging Eval Ecosystem

Eval-driven development is not just a best practice. It is becoming an industry. Q1 2026 saw significant market activity: Braintrust raised $80M at an $800M valuation, Langfuse was acquired by ClickHouse, and Humanloop joined Anthropic. The signal is clear: eval infrastructure is where the money flows.

The 2026 skill eval ecosystem map showing built-in frameworks, enterprise platforms, open source tools, and agent-specific benchmarks

Figure 5. The 2026 eval tool ecosystem, positioned by automation level and specialization. Built-in frameworks (skill-creator, Google ADK) offer the tightest integration. Enterprise platforms (Braintrust, W&B Weave) offer scale. Open source (DeepEval, Promptfoo) offers flexibility. Agent-specific benchmarks (SkillsBench, SWE-Bench) provide standardized comparison.

The ecosystem segments into four categories:

Built-in frameworks — Anthropic's skill-creator, Google's ADK eval harness, and OpenAI's evals framework. These integrate directly into the development workflow. The skill-creator is the most complete for skill-specific evaluation, with blind A/B comparison, iterative improvement, and an interactive review viewer.
Enterprise platforms — Braintrust, Arize AI, W&B Weave, Maxim AI. These target teams that need CI/CD integration, production monitoring, and compliance reporting. Braintrust's $800M valuation reflects the enterprise appetite for this category.
Open source — DeepEval (12,600+ GitHub stars), Promptfoo (Fortune 500 adoption), Langfuse, and Bloom (Anthropic's behavioral eval tool). These are the building blocks for teams that want to customize their eval pipeline.
Agent-specific benchmarks — SkillsBench (86 tasks, 7 models), SWE-Bench Verified, MCP-Bench. These provide standardized comparison points so teams can contextualize their results against the field.

"Eval-driven development is not optional. The era of shipping AI features built on prompt-engineering-by-anecdote is ending."

— Vercel Engineering Team, on their shift to systematic evaluation for AI features (2026)

Hamel Husain and Shreya Shankar — the most visible evangelists for eval-driven development — have trained over 2,000 PMs and engineers, including teams at OpenAI and Anthropic. Their O'Reilly book on AI evaluations is expected later in 2026. The trajectory is clear: eval expertise is becoming a core competency, not a nice-to-have.

Takeaway

Your Eval Checklist: Five Steps to Proof Before Ship

You do not need a $800M platform to start. The skill-creator framework runs locally. Here are five concrete steps to implement eval-driven development today:

Start with 2–3 test prompts. Write realistic prompts — the kind of thing a real user would actually say. Save them to evals/evals.json. Do not overthink this step. Three good prompts beat twenty weak ones.
Write assertions for the things that matter. What would make you confident the skill worked? A specific file format? A particular code pattern? Write those as expectations. Use code graders where possible, LLM-as-Judge where needed.
Run with-skill vs without-skill baselines. Always. Every time. The delta is the measurement. Without a baseline, you are measuring the model, not the skill.
Review the delta, not just the absolute score. A skill with 90% pass_rate sounds great until you learn the baseline was 88%. A skill with 65% pass_rate sounds mediocre until you learn the baseline was 20%. The delta tells the truth.
Iterate until the evidence says yes. Set an acceptance threshold. Run. Grade. Review. Improve. Repeat. The skill-creator's interactive viewer makes this a 10-minute feedback loop, not a multi-day exercise.

The bottom line

Eval-driven development is the difference between "I think this works" and "I can prove this works." In a world where 80% of enterprise AI investment fails to deliver value, proof is not pedantry. It is survival. The teams that build evaluation into their workflow will ship better AI. The ones that do not will ship faster — into production incidents.

What is your current eval process — ship-and-pray, or proof-before-ship?

Sources

Anthropic — Claude Code skill-creator framework with evaluation, benchmarking, and A/B testing (March 2026)
SkillsBench (arXiv:2602.12670) — Benchmarking Agent Skills across 86 tasks and 7 models: +16.2pp average pass rate improvement (March 2026)
Forrester Research — "Please Test Your AI Agents. Like, at All." Enterprise AI testing practices report (2026)
McKinsey & Company — State of AI 2026: 92% of organizations increasing AI investment, 40% scaling beyond pilots
BCG — RATE.ai framework: projects passing quality gates reach production 30% faster
Deloitte — State of AI in the Enterprise 2026: 94% of companies failing at AI production deployment
Judge Reliability Harness (arXiv:2603.05399) — Open-source library for stress-testing LLM judge reliability (2026)
Grading Scale Impact Study (arXiv:2601.03444) — 0-5 grading scale yields highest ICC between LLM judges and human evaluators (2026)
CNBC / Stanford AI Index — $684B AI investment in 2025, $547B+ failed to deliver expected value
EU AI Act — Full enforcement August 2, 2026 for high-risk AI quality management requirements
Hamel Husain & Shreya Shankar (Lenny's Newsletter) — Eval-driven development training: 2,000+ PMs and engineers including OpenAI and Anthropic teams
Braintrust — $80M Series B at $800M valuation for LLM evaluation platform (Q1 2026)
Google DORA Report — 2024 State of DevOps: AI adoption improves deployment frequency but can increase instability without strong foundations
ArXiv 2411.13768 — Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture
Harvard Business Review — How AI Fits into Lean Six Sigma (2023)