Evals
The eval is the product.
24 synthetic inpatient stays, each with an answer key written before its chart: codes, principal diagnosis, DRG, and whether a human was required. Every number below is recomputed from recorded runs by the same Python that gates CI. Intervals are 95% case-level bootstrap; at n = 24 they are honest, which means wide.
- Prompt version
- 2026-10-02.4-router-v2
- Operating threshold
- 0.80
- Models
- Haiku 4.5 (evidence) · Sonnet 5.5 (coder, critic, query)
- Recorded
- 2026-10-02
Agent vs one prompt
The baseline is the same Sonnet model given the whole chart and asked for codes: no tools, no evidence step, no critic, no validators. It routes on its own stated confidence and “needs review” flag. Same answer key, same threshold.
| At 0.80 | Agent | 95% CI | Baseline | 95% CI |
|---|---|---|---|---|
| Stays coded autonomously | 46% (11) | 33% (8) | ||
| Escapes (needed a human, got through) | 0 | 1 | ||
| Held needlessly | 5 | 9 | ||
| Code precision (autonomous) | 95.3% | 91.4%–98.8% | 92.3% | 87.5%–98.1% |
| Code recall (autonomous) | 94.2% | 90.4%–97.7% | 96.8% | 92.9%–100.0% |
| CC/MCC precision | 100% | 100%–100% | 100% | 100%–100% |
| CC/MCC recall | 100% | 100%–100% | 100% | 100%–100% |
| DRG match | 100% | 100%–100% | 100% | 100%–100% |
| Principal diagnosis correct | 82% | 55%–100% | 100% | 100%–100% |
| Over-coded (DRG above key) | 0% | 0%–0% | 0% | 0%–0% |
| Under-coded (DRG below key) | 0% | 0%–0% | 0% | 0%–0% |
| Invalid or non-billable codes | 0.0% | 0.0% | ||
| Cost per stay (all stays) | $0.096 | $0.011 | ||
| Latency p50 / p95 | 41s / 52s | 5s / 10s |
On the stays each system finalizes on its own, both get the DRG right; these charts are short and the traps are about documentation, not grouping. The difference is in what gets through without a human, how much reaches autonomy at all, and whether each code can be explained with a verbatim quote.
| Threshold | Agent | Baseline | Agent v1 |
|---|---|---|---|
| 0.00 | 100% | 100% | 100% |
| 0.05 | 100% | 100% | 100% |
| 0.10 | 96% | 100% | 96% |
| 0.15 | 96% | 100% | 92% |
| 0.20 | 92% | 100% | 88% |
| 0.25 | 88% | 88% | 79% |
| 0.30 | 83% | 75% | 71% |
| 0.35 | 79% | 54% | 63% |
| 0.40 | 79% | 42% | 63% |
| 0.45 | 67% | 42% | 54% |
| 0.50 | 54% | 42% | 42% |
| 0.55 | 54% | 42% | 42% |
| 0.60 | 54% | 42% | 42% |
| 0.65 | 46% | 42% | 38% |
| 0.70 | 46% | 42% | 38% |
| 0.75 | 46% | 33% | 38% |
| 0.80 | 46% | 33% | 33% |
| 0.85 | 42% | 8% | 33% |
| 0.90 | 29% | 0% | 17% |
| 0.95 | 4% | 0% | 4% |
| 1.00 | 0% | 0% | 0% |
| Threshold | Agent | Baseline | Agent v1 |
|---|---|---|---|
| 0.00 | 100% | 100% | 100% |
| 0.05 | 100% | 100% | 100% |
| 0.10 | 88% | 100% | 88% |
| 0.15 | 88% | 100% | 88% |
| 0.20 | 88% | 100% | 88% |
| 0.25 | 75% | 88% | 75% |
| 0.30 | 63% | 63% | 50% |
| 0.35 | 50% | 25% | 50% |
| 0.40 | 50% | 13% | 50% |
| 0.45 | 25% | 13% | 38% |
| 0.50 | 0% | 13% | 13% |
| 0.55 | 0% | 13% | 13% |
| 0.60 | 0% | 13% | 13% |
| 0.65 | 0% | 13% | 13% |
| 0.70 | 0% | 13% | 13% |
| 0.75 | 0% | 13% | 13% |
| 0.80 | 0% | 13% | 13% |
| 0.85 | 0% | 0% | 13% |
| 0.90 | 0% | 0% | 0% |
| 0.95 | 0% | 0% | 0% |
| 1.00 | 0% | 0% | 0% |
What failed
v1 let the model decide what blocks autonomy.
In v1 every issue carried a blocks_autonomy boolean set by the coder or critic. Error analysis on the first full run found the model was noisy about it in both directions:
- It over-held clean stays, blocking on a non-specific knee-osteoarthritis code, on a stale copy-forwarded line that newer notes had superseded, and on a “questioned” principal whose alternative grouped to the same DRG.
- It let the heart-failure stay escape: a creatinine rise no provider named was flagged as a query opportunity, but not as blocking.
v2 splits the job. The model labels each issue’s impact on the claim (principal, severity, none). router.py decides with rules a reviewer can read: block on provider questions that can move the claim, on critic doubts about codes that set the DRG, on a questioned principal only if it regroups, and on any hospital-acquired condition coded POA = N. Because routing is plain code over recorded labels, that last rule was applied to every recorded run with no new model calls.
| v1 | v2 | |
|---|---|---|
| Coded autonomously | 33% | 46% |
| Held needlessly | 9 | 5 |
| Escapes | 1 | 0 |
| Code precision | 93.7% | 95.3% |
$ uv run python gate.py --runs agent-v1 ✓ code precision (auto) 93.7% >= 92% ✓ DRG match (auto) 100% >= 95% ✓ over-coding rate 0% <= 0% ✗ escape rate 12.5% <= 0% exit 1 $ uv run python gate.py ✓ … all five checks pass exit 0
The CI gate re-scores committed runs on every PR, with no API calls: a worse recording, an edited answer key, or a grouper change that moves a DRG fails the build.
Where the remaining errors are
Code-level misses grouped by the trap planted in the chart. Most are secondary-code choices that don’t move the DRG (a status Z-code, a more specific organism), which is why precision is below DRG match.
What a stay costs
Average per stay, from recorded token usage at list prices. Validators rejected and re-prompted 10 outputs across the run (mostly paraphrased quotes); none reached the result.
| Stage | Model | Tokens in / out | Time | Cost |
|---|---|---|---|---|
| evidence | haiku-4-5 | 5.0k / 3.0k | 18.1s | $0.020 |
| coder | sonnet-5-5 | 18.0k / 1.6k | 13.8s | $0.052 |
| critic | sonnet-5-5 | 6.5k / 0.8k | 6.3s | $0.021 |
| query | sonnet-5-5 | 0.6k / 0.2k | 3.9s | $0.003 |
| Total per stay | $0.096 | |||
Queries are drafted only for held stays, so their cost is spread across all stays. Next lever: prompt-cache the chart across the coder and critic calls, which share it as the largest input.
How to read these numbers
- • Charts are synthetic and short (500–1,000 words; real stays run around 50,000).
- • Answer keys were written by the author before each chart, not by credentialed coders.
- • Charts were written by Claude in a separate session; the agent runs on Haiku and Sonnet. Same family, so some circularity is possible.
- • DRG weights and the CC/MCC list are illustrative; codes are real FY2027 ICD-10-CM.
- • One recorded run per stay. Run-to-run variance is not yet measured; that’s the next eval.
- • Re-run on Akasa’s gold set, these numbers would mean something. The harness is the point.