Get in touch
Menu

Evals · Demo suite

Demo figures for a fictional health-system suite.

Twelve agents on one proprietary-shaped suite. Hover any mark below for the model, the harness, and the environment behind it; click to pin it across every figure. Every number is a demo value for a fictional company, Harborlight Health, never a measured client result. The same specimen name appears in Publications anatomy drafts; those are seating and population method pages with public-eval stand-ins, not these twelve-agent suite plots.

Back to Evals

We hold no HIPAA attestation of our own — we work inside your governed environment, under your BAA.

Figures

Score, cost, tokens, and speed - one suite, twelve agents

The side nav jumps between metrics. The strip above the figures names whichever agent is under the pointer (or pinned). Colour groups environments; glyph shape groups harnesses; every mark carries its id.

demo · illustrative specimen · not MeasuredHarborlight intake-to-discharge suite · demo v0.3 · 1,240 cases · 9,860 rubric points · as of 2026-10-14

A proprietary-shaped evaluation suite for one fictional regional health system: cases drawn from its own intake, referral, discharge, and chart-question workflows, scored against rubric points its own clinicians wrote. Every row is a full agent: the model, the harness around it, and the environment it ran in.

Harborlight Health: fictional regional health system · specimen · illustrative · not a client. 12 agents across 6 models, 5 harnesses and 4 environments.

A-01M-01H-04E-02Model · Harness · Environment

A-01top adjusted score · hover any mark

Model
M-01 Hosted frontier model, reasoning modeA hosted frontier model with extended reasoning on. Highest ceiling, longest answers, most tokens.
Harness
H-04 Planner and verifierTwo roles: one drafts, a second checks the draft against the retrieved sources and the rubric before it is returned.
Environment
E-02 Client cloud tenancy, hosted endpointA model endpoint inside the client's own cloud tenancy, under their access controls.
Adjusted
71.4%
Raw
74.9%
Cost / task
$0.42
Output tokens
30k
Answer length
4,200 ch
Time / task
61s
Run
Sep 2026

demo · illustrative specimen · not Measured Harborlight Health · fictional regional health system · specimen · illustrative · not a client

Score

Fig. 01Length-adjusted score

A-01: 71.4% length-adjusted score. Model M-01, Hosted frontier model, reasoning mode; harness H-04, Planner and verifier; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.71.4%A-01A-02: 69.8% length-adjusted score. Model M-01, Hosted frontier model, reasoning mode; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.69.8%A-02A-04: 66.1% length-adjusted score. Model M-02, Hosted frontier model, standard mode; harness H-02, Retrieval over the chart; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.66.1%A-04A-08: 64.5% length-adjusted score. Model M-04, Open-weights model, 70B class; harness H-04, Planner and verifier; environment E-03, On-prem department server. Demo value, not measured.64.5%A-08A-03: 63.2% length-adjusted score. Model M-01, Hosted frontier model, reasoning mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.63.2%A-03A-06: 61.9% length-adjusted score. Model M-03, Hosted mid-size model; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.61.9%A-06A-09: 60.2% length-adjusted score. Model M-04, Open-weights model, 70B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.60.2%A-09A-05: 58.7% length-adjusted score. Model M-02, Hosted frontier model, standard mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.58.7%A-05A-11: 55.6% length-adjusted score. Model M-05, Open-weights model, 8B class; harness H-05, Multi-agent with a human gate; environment E-04, On-prem clinic edge device. Demo value, not measured.55.6%A-11A-07: 52.4% length-adjusted score. Model M-03, Hosted mid-size model; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.52.4%A-07A-10: 49.8% length-adjusted score. Model M-05, Open-weights model, 8B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.49.8%A-10A-12: 46.3% length-adjusted score. Model M-06, Clinical fine-tune, 3B class; harness H-01, Single prompt; environment E-04, On-prem clinic edge device. Demo value, not measured.46.3%A-12
Length-adjusted: answers longer than the receiving clinician's limit (demo: 2,000 characters) lose rubric points; shorter ones gain. Raw: the share of rubric points earned with no length rule. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentLength-adjustedRaw
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint71.4%74.9%
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint69.8%72.1%
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input63.2%67.5%
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint66.1%66.9%
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input58.7%60.3%
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint61.9%62.4%
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input52.4%53.0%
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server64.5%68.8%
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server60.2%61.0%
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server49.8%50.1%
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device55.6%57.9%
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device46.3%46.0%

Fig. 02Length-adjusted score vs cost per task

A-01: 71.4% adjusted · $0.42/task. Model M-01, Hosted frontier model, reasoning mode; harness H-04, Planner and verifier; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-01A-02: 69.8% adjusted · $0.31/task. Model M-01, Hosted frontier model, reasoning mode; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-02A-03: 63.2% adjusted · $0.12/task. Model M-01, Hosted frontier model, reasoning mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-03A-04: 66.1% adjusted · $0.061/task. Model M-02, Hosted frontier model, standard mode; harness H-02, Retrieval over the chart; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-04A-05: 58.7% adjusted · $0.034/task. Model M-02, Hosted frontier model, standard mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-05A-06: 61.9% adjusted · $0.019/task. Model M-03, Hosted mid-size model; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-06A-07: 52.4% adjusted · $0.0062/task. Model M-03, Hosted mid-size model; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-07A-08: 64.5% adjusted · $0.087/task. Model M-04, Open-weights model, 70B class; harness H-04, Planner and verifier; environment E-03, On-prem department server. Demo value, not measured.A-08A-09: 60.2% adjusted · $0.028/task. Model M-04, Open-weights model, 70B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-09A-10: 49.8% adjusted · $0.0048/task. Model M-05, Open-weights model, 8B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-10A-11: 55.6% adjusted · $0.011/task. Model M-05, Open-weights model, 8B class; harness H-05, Multi-agent with a human gate; environment E-04, On-prem clinic edge device. Demo value, not measured.A-11A-12: 46.3% adjusted · $0.0031/task. Model M-06, Clinical fine-tune, 3B class; harness H-01, Single prompt; environment E-04, On-prem clinic edge device. Demo value, not measured.A-12
Cost per task is the average over the suite. For hosted endpoints it is the invoice; for on-prem environments it is amortized hardware time at the organization's own rate. Log scale. The frontier joins the agents no other agent beats on both axes. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentLength-adjustedRawCost per task
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint71.4%74.9%$0.42
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint69.8%72.1%$0.31
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input63.2%67.5%$0.12
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint66.1%66.9%$0.061
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input58.7%60.3%$0.034
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint61.9%62.4%$0.019
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input52.4%53.0%$0.0062
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server64.5%68.8%$0.087
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server60.2%61.0%$0.028
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server49.8%50.1%$0.0048
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device55.6%57.9%$0.011
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device46.3%46.0%$0.0031

Fig. 03Score vs answer length

A-01: 71.4% adjusted, 74.9% raw · 4,200 characters. Model M-01, Hosted frontier model, reasoning mode; harness H-04, Planner and verifier; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-01A-02: 69.8% adjusted, 72.1% raw · 3,600 characters. Model M-01, Hosted frontier model, reasoning mode; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-02A-03: 63.2% adjusted, 67.5% raw · 4,700 characters. Model M-01, Hosted frontier model, reasoning mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-03A-04: 66.1% adjusted, 66.9% raw · 2,300 characters. Model M-02, Hosted frontier model, standard mode; harness H-02, Retrieval over the chart; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-04A-05: 58.7% adjusted, 60.3% raw · 2,500 characters. Model M-02, Hosted frontier model, standard mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-05A-06: 61.9% adjusted, 62.4% raw · 2,100 characters. Model M-03, Hosted mid-size model; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-06A-07: 52.4% adjusted, 53.0% raw · 1,900 characters. Model M-03, Hosted mid-size model; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-07A-08: 64.5% adjusted, 68.8% raw · 4,100 characters. Model M-04, Open-weights model, 70B class; harness H-04, Planner and verifier; environment E-03, On-prem department server. Demo value, not measured.A-08A-09: 60.2% adjusted, 61.0% raw · 2,400 characters. Model M-04, Open-weights model, 70B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-09A-10: 49.8% adjusted, 50.1% raw · 1,800 characters. Model M-05, Open-weights model, 8B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-10A-11: 55.6% adjusted, 57.9% raw · 2,900 characters. Model M-05, Open-weights model, 8B class; harness H-05, Multi-agent with a human gate; environment E-04, On-prem clinic edge device. Demo value, not measured.A-11A-12: 46.3% adjusted, 46.0% raw · 1,400 characters. Model M-06, Clinical fine-tune, 3B class; harness H-01, Single prompt; environment E-04, On-prem clinic edge device. Demo value, not measured.A-12
Each agent at its mean answer length: the faded mark is the raw score, the solid one the length-adjusted score. Answers past 2,000 characters lose points. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentLength-adjustedRawMean answer (characters)
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint71.4%74.9%4,200
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint69.8%72.1%3,600
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input63.2%67.5%4,700
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint66.1%66.9%2,300
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input58.7%60.3%2,500
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint61.9%62.4%2,100
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input52.4%53.0%1,900
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server64.5%68.8%4,100
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server60.2%61.0%2,400
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server49.8%50.1%1,800
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device55.6%57.9%2,900
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device46.3%46.0%1,400

Fig. 04Score by use case, type and difficulty

Rubric slice
Overall (raw)74.9%72.1%68.8%67.5%66.9%62.4%61%60.3%57.9%53%50.1%46%
Use case · Intake triage73%70%67%64%65%60%59%57%55%50%48%43%
Use case · Discharge summary79%76%73%72%71%66%65%64%64%57%56%52%
Use case · Referral letter77%74%71%70%69%64%63%62%60%55%52%53%
Use case · Chart question79%76%73%66%71%66%65%58%56%51%54%38%
Type · Good faith81%78%75%74%73%68%67%66%64%59%56%52%
Type · Adversarial73%63%67%58%58%53%52%51%50%44%35%31%
Difficulty · Typical85%82%79%78%77%72%71%70%68%63%60%56%
Difficulty · Difficult68%61%62%56%56%51%50%49%46%42%34%30%
Raw score, no length rule, per rubric slice of the suite. Best overall agent first. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentRawIntake triageDischarge summaryReferral letterChart questionGood faithAdversarialTypicalDifficult
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint74.9%73%79%77%79%81%73%85%68%
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint72.1%70%76%74%76%78%63%82%61%
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input67.5%64%72%70%66%74%58%78%56%
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint66.9%65%71%69%71%73%58%77%56%
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input60.3%57%64%62%58%66%51%70%49%
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint62.4%60%66%64%66%68%53%72%51%
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input53.0%50%57%55%51%59%44%63%42%
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server68.8%67%73%71%73%75%67%79%62%
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server61.0%59%65%63%65%67%52%71%50%
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server50.1%48%56%52%54%56%35%60%34%
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device57.9%55%64%60%56%64%50%68%46%
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device46.0%43%52%53%38%52%31%56%30%

Token usage

Fig. 05Output tokens per task

A-12: 1.6k output tokens per task (answer 1.6k). Model M-06, Clinical fine-tune, 3B class; harness H-01, Single prompt; environment E-04, On-prem clinic edge device. Demo value, not measured.1.6kA-12A-07: 2.2k output tokens per task (answer 2.2k). Model M-03, Hosted mid-size model; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.2.2k2.2kA-07A-10: 2.4k output tokens per task (answer 2.4k). Model M-05, Open-weights model, 8B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.2.4k2.4kA-10A-05: 2.6k output tokens per task (answer 2.6k). Model M-02, Hosted frontier model, standard mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.2.6k2.6kA-05A-09: 3.6k output tokens per task (answer 3.0k, reasoning 600). Model M-04, Open-weights model, 70B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.3.0k3.6kA-09A-04: 3.9k output tokens per task (answer 3.1k, reasoning 800). Model M-02, Hosted frontier model, standard mode; harness H-02, Retrieval over the chart; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.3.1k3.9kA-04A-06: 4.5k output tokens per task (answer 2.9k, reasoning 1.6k). Model M-03, Hosted mid-size model; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.2.9k4.5kA-06A-11: 8.0k output tokens per task (answer 3.8k, reasoning 4.2k). Model M-05, Open-weights model, 8B class; harness H-05, Multi-agent with a human gate; environment E-04, On-prem clinic edge device. Demo value, not measured.3.8k4.2k8.0kA-11A-03: 14k output tokens per task (answer 4.8k, reasoning 9.0k). Model M-01, Hosted frontier model, reasoning mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.4.8k9.0k14kA-03A-08: 19k output tokens per task (answer 5.2k, reasoning 14k). Model M-04, Open-weights model, 70B class; harness H-04, Planner and verifier; environment E-03, On-prem department server. Demo value, not measured.5.2k14k19kA-08A-02: 22k output tokens per task (answer 5.4k, reasoning 17k). Model M-01, Hosted frontier model, reasoning mode; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.5.4k17k22kA-02A-01: 30k output tokens per task (answer 6.1k, reasoning 24k). Model M-01, Hosted frontier model, reasoning mode; harness H-04, Planner and verifier; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.6.1k24k30kA-01
Output tokens used to run one task, split into answer and reasoning tokens. Lowest first. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentAnswer tokensReasoning tokensLength-adjusted
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint6,10024,00071.4%
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint5,40017,00069.8%
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input4,8009,00063.2%
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint3,10080066.1%
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input2,600058.7%
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint2,9001,60061.9%
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input2,200052.4%
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server5,20014,00064.5%
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server3,00060060.2%
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server2,400049.8%
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device3,8004,20055.6%
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device1,600046.3%

Cost

Fig. 06Cost per task

A-12: $0.0031 cost per task (input $0.0012, cache write $0.0001, cache hit $0.0001, answer $0.0017). Model M-06, Clinical fine-tune, 3B class; harness H-01, Single prompt; environment E-04, On-prem clinic edge device. Demo value, not measured.$0.0031A-12A-10: $0.0048 cost per task (input $0.0010, cache write $0.0004, cache hit $0.0008, answer $0.0026). Model M-05, Open-weights model, 8B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.$0.0048A-10A-07: $0.0062 cost per task (input $0.0012, cache write $0.0001, cache hit $0.0001, answer $0.0048). Model M-03, Hosted mid-size model; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.$0.0062A-07A-11: $0.011 cost per task (input $0.0022, cache write $0.0010, cache hit $0.0017, reasoning $0.0029, answer $0.0032). Model M-05, Open-weights model, 8B class; harness H-05, Multi-agent with a human gate; environment E-04, On-prem clinic edge device. Demo value, not measured.$0.011A-11A-06: $0.019 cost per task (input $0.0019, cache write $0.0008, cache hit $0.0015, reasoning $0.0047, answer $0.010). Model M-03, Hosted mid-size model; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.$0.019A-06A-09: $0.028 cost per task (input $0.0057, cache write $0.0025, cache hit $0.0044, reasoning $0.0023, answer $0.013). Model M-04, Open-weights model, 70B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.$0.028A-09A-05: $0.034 cost per task (input $0.0064, cache write $0.0004, cache hit $0.0008, answer $0.027). Model M-02, Hosted frontier model, standard mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.$0.034A-05A-04: $0.061 cost per task (input $0.0060, cache write $0.0027, cache hit $0.0047, reasoning $0.0088, answer $0.039). Model M-02, Hosted frontier model, standard mode; harness H-02, Retrieval over the chart; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.$0.039$0.061A-04A-08: $0.087 cost per task (input $0.018, cache write $0.0078, cache hit $0.014, reasoning $0.030, answer $0.018). Model M-04, Open-weights model, 70B class; harness H-04, Planner and verifier; environment E-03, On-prem department server. Demo value, not measured.$0.030$0.087A-08A-03: $0.12 cost per task (input $0.022, cache write $0.0013, cache hit $0.0026, reasoning $0.055, answer $0.039). Model M-01, Hosted frontier model, reasoning mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.$0.055$0.039$0.12A-03A-02: $0.31 cost per task (input $0.031, cache write $0.014, cache hit $0.024, reasoning $0.15, answer $0.092). Model M-01, Hosted frontier model, reasoning mode; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.$0.031$0.15$0.092$0.31A-02A-01: $0.42 cost per task (input $0.042, cache write $0.018, cache hit $0.032, reasoning $0.20, answer $0.12). Model M-01, Hosted frontier model, reasoning mode; harness H-04, Planner and verifier; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.$0.042$0.032$0.20$0.12$0.42A-01
Average cost per task, split into input, cache write, cache hit, reasoning and answer tokens. Cost per task is the average over the suite. For hosted endpoints it is the invoice; for on-prem environments it is amortized hardware time at the organization's own rate. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentInputCache writeCache hitReasoningAnswerPer taskFull suite
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint$0.042$0.018$0.032$0.20$0.12$0.42$521
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint$0.031$0.014$0.024$0.15$0.092$0.31$384
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input$0.022$0.0013$0.0026$0.055$0.039$0.12$149
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint$0.0060$0.0027$0.0047$0.0088$0.039$0.061$76
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input$0.0064$0.0004$0.0008$0.0000$0.027$0.034$42
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint$0.0019$0.0008$0.0015$0.0047$0.010$0.019$24
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input$0.0012$0.0001$0.0001$0.0000$0.0048$0.0062$8
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server$0.018$0.0078$0.014$0.030$0.018$0.087$108
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server$0.0057$0.0025$0.0044$0.0023$0.013$0.028$35
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server$0.0010$0.0004$0.0008$0.0000$0.0026$0.0048$6
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device$0.0022$0.0010$0.0017$0.0029$0.0032$0.011$14
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device$0.0012$0.0001$0.0001$0.0000$0.0017$0.0031$4

Speed

Fig. 07Seconds per task

A-12: 4s seconds per task. Model M-06, Clinical fine-tune, 3B class; harness H-01, Single prompt; environment E-04, On-prem clinic edge device. Demo value, not measured.A-12H-01 · E-044sA-07: 6s seconds per task. Model M-03, Hosted mid-size model; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-07H-01 · E-016sA-10: 8s seconds per task. Model M-05, Open-weights model, 8B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-10H-02 · E-038sA-05: 9s seconds per task. Model M-02, Hosted frontier model, standard mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-05H-01 · E-019sA-04: 14s seconds per task. Model M-02, Hosted frontier model, standard mode; harness H-02, Retrieval over the chart; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-04H-02 · E-0214sA-06: 17s seconds per task. Model M-03, Hosted mid-size model; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-06H-03 · E-0217sA-03: 29s seconds per task. Model M-01, Hosted frontier model, reasoning mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-03H-01 · E-0129sA-09: 31s seconds per task. Model M-04, Open-weights model, 70B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-09H-02 · E-0331sA-11: 42s seconds per task. Model M-05, Open-weights model, 8B class; harness H-05, Multi-agent with a human gate; environment E-04, On-prem clinic edge device. Demo value, not measured.A-11H-05 · E-0442sA-02: 48s seconds per task. Model M-01, Hosted frontier model, reasoning mode; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-02H-03 · E-0248sA-01: 61s seconds per task. Model M-01, Hosted frontier model, reasoning mode; harness H-04, Planner and verifier; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-01H-04 · E-0261sA-08: 95s seconds per task. Model M-04, Open-weights model, 70B class; harness H-04, Planner and verifier; environment E-03, On-prem department server. Demo value, not measured.A-08H-04 · E-0395s
Median wall-clock time from request to final answer, including every tool call and hand-off in the harness. Fastest first. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentSeconds per taskLength-adjusted
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint6171.4%
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint4869.8%
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input2963.2%
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint1466.1%
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input958.7%
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint1761.9%
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input652.4%
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server9564.5%
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server3160.2%
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server849.8%
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device4255.6%
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device446.3%

Score vs run date

Fig. 08Length-adjusted score by run date

A-01: 71.4% score · run Sep 2026. Model M-01, Hosted frontier model, reasoning mode; harness H-04, Planner and verifier; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-01A-02: 69.8% score · run Aug 2026. Model M-01, Hosted frontier model, reasoning mode; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-02A-03: 63.2% score · run Jul 2026. Model M-01, Hosted frontier model, reasoning mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-03A-04: 66.1% score · run Aug 2026. Model M-02, Hosted frontier model, standard mode; harness H-02, Retrieval over the chart; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-04A-05: 58.7% score · run Jun 2026. Model M-02, Hosted frontier model, standard mode; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-05A-06: 61.9% score · run Sep 2026. Model M-03, Hosted mid-size model; harness H-03, Tool-using agent, EHR read tools; environment E-02, Client cloud tenancy, hosted endpoint. Demo value, not measured.A-06A-07: 52.4% score · run May 2026. Model M-03, Hosted mid-size model; harness H-01, Single prompt; environment E-01, Hosted API, de-identified input. Demo value, not measured.A-07A-08: 64.5% score · run Sep 2026. Model M-04, Open-weights model, 70B class; harness H-04, Planner and verifier; environment E-03, On-prem department server. Demo value, not measured.A-08A-09: 60.2% score · run Aug 2026. Model M-04, Open-weights model, 70B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-09A-10: 49.8% score · run Jul 2026. Model M-05, Open-weights model, 8B class; harness H-02, Retrieval over the chart; environment E-03, On-prem department server. Demo value, not measured.A-10A-11: 55.6% score · run Oct 2026. Model M-05, Open-weights model, 8B class; harness H-05, Multi-agent with a human gate; environment E-04, On-prem clinic edge device. Demo value, not measured.A-11A-12: 46.3% score · run Jun 2026. Model M-06, Clinical fine-tune, 3B class; harness H-01, Single prompt; environment E-04, On-prem clinic edge device. Demo value, not measured.A-12
When each agent configuration was last run against this version of the suite. A suite is re-run whenever the model, the harness or the environment changes. Source: Harborlight Health (fictional regional health system · specimen · illustrative · not a client) · demo · illustrative specimen · not Measured.
Show the data
AgentModelHarnessEnvironmentRun dateLength-adjusted
A-01M-01 Hosted frontier model, reasoning modeH-04 Planner and verifierE-02 Client cloud tenancy, hosted endpoint2026-09-1871.4%
A-02M-01 Hosted frontier model, reasoning modeH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint2026-08-3069.8%
A-03M-01 Hosted frontier model, reasoning modeH-01 Single promptE-01 Hosted API, de-identified input2026-07-2263.2%
A-04M-02 Hosted frontier model, standard modeH-02 Retrieval over the chartE-02 Client cloud tenancy, hosted endpoint2026-08-1266.1%
A-05M-02 Hosted frontier model, standard modeH-01 Single promptE-01 Hosted API, de-identified input2026-06-1058.7%
A-06M-03 Hosted mid-size modelH-03 Tool-using agent, EHR read toolsE-02 Client cloud tenancy, hosted endpoint2026-09-0261.9%
A-07M-03 Hosted mid-size modelH-01 Single promptE-01 Hosted API, de-identified input2026-05-2852.4%
A-08M-04 Open-weights model, 70B classH-04 Planner and verifierE-03 On-prem department server2026-09-2564.5%
A-09M-04 Open-weights model, 70B classH-02 Retrieval over the chartE-03 On-prem department server2026-08-0560.2%
A-10M-05 Open-weights model, 8B classH-02 Retrieval over the chartE-03 On-prem department server2026-07-0149.8%
A-11M-05 Open-weights model, 8B classH-05 Multi-agent with a human gateE-04 On-prem clinic edge device2026-10-1455.6%
A-12M-06 Clinical fine-tune, 3B classH-01 Single promptE-04 On-prem clinic edge device2026-06-2046.3%

Demo data for a fictional specimen organization, derived from the shape of real evaluation work, never a client's numbers. A client's suite shows their cases, their rubric slices, and their agents, in their environment. We hold no HIPAA attestation of our own — we work inside your governed environment, under your BAA.

Ask about a suite for your workflows.

The demo is a specimen. An engagement builds the cases and rubric from your own work, runs in your environment, and hands over the set, the harness, and the thresholds.

Ask about custom evalsBack to Evals

How can we help?

Name the problem you're working on. Shaun typically replies within two working days, by email, with the consultants in the practice who have solved that shape of problem before.

Topic (optional)
Prefer to talk? Schedule a 30-minute call