How it works
An autonomous coder is only as good as its off switch.
On October 2, 2026, Akasa announced autonomous coding for inpatient stays, with a citation for every code, a confidence score, and a threshold the health system sets. I built Threshold the same day as my application for Software Engineer, Applied AI. It isn’t a copy of that product. It’s the part I’d want to own behind it: the harness that tells you which stays to trust, the router that sends the rest to a coder with a reason, and the evals that keep both honest as prompts, models and code sets change.
The job description asks for someone who can design agentic systems, run offline and online evals with statistics, and balance coding guidelines, user feedback and auditability. That’s what this demo tries to show, on 24 synthetic charts, in one day.
Principles
- 01
Every code points at a sentence.
Quotes are validated against the chart by character offset. A paraphrase is rejected and the model is asked again.
- 02
Only real codes exist.
The coder looks codes up in the FY2027 ICD-10-CM table through tools. A header or invented code is sent back with its billable children.
- 03
Models label; code decides.
LLMs read, choose codes, argue against them, and write queries. Ordering documents, DRG grouping, routing and linting are deterministic and unit-tested.
- 04
Accuracy in both directions.
Over-coding is tracked as its own metric, the critic can only remove codes or lower confidence, and queries are linted so they can't lead a physician or mention money.
- 05
Escapes are the number that matters.
Autonomy is only worth having if the stays that need a human still reach one. The threshold is chosen against escapes, not against accuracy.
Architecture
chart ─▶ triage (code) ─▶ evidence (Haiku 4.5) ── quotes validated by offset
│
▼
coder (Sonnet 5.5) ◀─▶ index_lookup / tabular_lookup
│ (FY2027 ICD-10-CM, SQLite FTS5)
▼ ── codes validated: billable, cited
critic (Sonnet 5.5) ── can remove codes, lower confidence
│
▼
grouper (code) ─▶ router (code) ─▶ query (Sonnet + linter)
│
▼
JSONL trace ─▶ eval harness ─▶ CI gate ─▶ this site (replays)Why split it into agents? Each stage has one job and one typed output, so each can be validated, priced and evaluated on its own. Extraction is cheap pattern work and runs on Haiku. Coding needs guideline judgment and tools, so it runs on Sonnet. The critic is a second, adversarial read: it was told to refute, not agree. The single-prompt baseline on the evals page is there so this split has to earn its cost.
Why replays instead of a live button? Recorded runs are deterministic, free to serve, and safe on a public URL. The trace schema is the same one a live run streams, so a rate-limited live mode (FastAPI with SSE on Fly.io) is a deployment step, not a rewrite.
The eval
Each of the 24 charts started as an answer key: principal diagnosis, secondary codes with POA, DRG, and whether a human must look. Then the chart was written to produce it, with traps planted on purpose: a late CT addendum that contradicts the first read, “probable sepsis” at discharge (codable inpatient), sepsis documented then ruled out by a different physician, a lab-only kidney injury, malnutrition only in a dietitian’s note, a hospital-acquired pressure ulcer, copy-forwarded progress notes.
Metrics are micro-averaged at the code level with case-level bootstrap intervals, plus DRG match, principal accuracy, CC/MCC precision and recall, over- and under-coding against the key’s DRG, citation validity, query compliance, cost and latency. The reviewer accept/reject buttons on each stay show where the online signal would come from: accept rate by confidence bin is how the threshold gets recalibrated in production.
What failed
Letting the model decide what blocks. v1 asked the model to mark each issue as blocking. It over-held clean stays and let one through that needed a query. At the 0.80 threshold, v1 coded 33% of stays autonomously with 1 escape; v2, which has the model label impact and lets code decide, codes 46% with 0. The CI gate fails v1 on escapes. Details on the evals page.
Parallel tool calls on one SQLite connection. The first full run lost five stays to “bad parameter or other API misuse”: Pydantic AI runs sync tools in worker threads, and the code index shared a connection. A lock fixed it; a threaded test keeps it fixed.
Paraphrased citations. The extraction model paraphrased quotes often enough that the validator re-prompted 10 outputs across the run. None reached the site, which is the point of validating, but it costs latency. Next step: ask for document offsets directly, or use the API’s native citations.
Agents disagree with the key on minor codes. Most remaining misses are status Z-codes and organism specificity that don’t move the DRG. A real gold set would need adjudication rules for those, and probably a separate “claim-affecting” precision metric.
Limits, plainly
- • Synthetic, short charts. Real stays are about 50,000 words across 60 to 100 documents.
- • Answer keys written by me, not credentialed coders, and not adjudicated.
- • The DRG grouper covers seven medical families with illustrative weights and a simplified CC/MCC list.
- • One run per stay; no variance estimate yet.
- • No CPT anywhere (AMA-licensed). No patient data anywhere.
- • Not affiliated with or endorsed by Akasa. Akasa’s product details here come from its public site and press.
What I’d do next
- 1. Live mode: FastAPI + SSE on Fly.io, three curated charts, rate-limited, with “this run vs recorded” agreement shown.
- 2. Variance: five runs per stay, and route on agreement across runs as a second confidence signal.
- 3. A coder-reviewed gold subset with inter-rater agreement reported before adjudication.
- 4. FY regression gate: replay the set whenever the code table changes (FY2027 took effect Oct 1).
- 5. Calibration: fit isotonic regression on accept/reject feedback, and report expected calibration error.
Built by Hafsa Usmani. Source, tests and recorded runs are on GitHub.