Workspace
Production routes
See what is serving, what it costs, and whether it still clears your test.
Workspace access
Create a private workspace.
No account is required. One private key links this browser to encrypted route metadata. Prompts, inputs, outputs, cases, responses, and provider keys stay local.
Workspace ID checking…. The SDK must show this same ID.
python3 -m evalt connectRouteQualityCost / 1KServingLast checked
No account requiredCreate a private key to sync sanitized route metadata.
Recorded example
security-answers
Answer customer security questions from approved company evidence without inventing claims.
Selected configurationGPT OSS 20B · medium
Observed final test12 / 12Quality floor95%Production cost$2.13 / 1KPromptStarting prompt
Selected only after the complete batch settled. A cheaper configuration was tested and failed the frozen quality gate.
Cost frontier
95% requiredWhy this route is serving
Higher production costLower production cost
Frozen contract v1
Human-reviewed structureEvidence before averages
Every candidate saw the same 30 systems. Six separate final-test systems were run twice.
Decision history
Every route change leaves a receipt.
Promotions, rejections, price changes, and manual approvals stay attached to this task.
GPT OSS 20B · medium promoted
100% observed final test · $2.13 / 1K calls · starting prompt retained
GPT OSS 20B · low rejected
Cheaper, but 83.3% missed the frozen 95% gate.
Prompt rewrite tested
Improved one DeepSeek lane from 50% to 66.7%; not promoted.
Contract v1 frozen
30 cases · repeated final test · $0.18 test cap.
Production handoff
Your code names the task, not today’s model.
The SDK executes the promoted configuration and keeps feedback isolated to this route.
Python
from evalt import Evalt
evalt = Evalt()
answer = evalt.run(
security_prompt,
customer_question,
route="security-answers",
target_accuracy=0.95,
)
print(answer.content)The selected prompt, model, reasoning effort, and price snapshot are versioned together.
Accepted and corrected outputs become evidence only for security-answers.
A challenger cannot replace the serving route unless it clears the same frozen contract.
Local SDK route
Connectedroute
Operational evidence synchronized from your local Evalt SDK.
Selected configurationWaiting for a tested route
Observed final test—Quality floor—Production cost—Production calls0
Progress appears here while the local tournament runs. Dashboard downtime never blocks the SDK.
What the optimizer is doing
Workspace activity
Progress without raw traces.
Follow test design, model screening, prompt trials, production calls, and route decisions without uploading customer content.
security-answers promoted GPT OSS 20B · medium
The candidate passed 12/12 final-test executions at $2.13 per 1,000 calls.
Cheaper low-reasoning route rejected
83.3% was below the route’s 95% requirement.
Frozen contract approved
30 cases across routine, complex, and adversarial work.
Open source on GitHub
Hit your quality bar. Cut the AI cost.
Turn a recurring AI task into a tested production route. On the first call, Evalt builds and judges the test, searches prompts, few-shot examples, models, and reasoning levels, then serves the cheapest configuration proven to clear your quality bar.
Python packageevaltv0.10.20 · Python 3.10+
python3 -m pip install --upgrade https://evalt.onrender.com/python-sdk/dist/evalt-0.10.20-py3-none-any.whl
Quality target 95%Tested 1 of 8Cheapest pass Searching
GPT-5.6 Luna · low reasoning
Overall: 5/6 systems · 83.3% · configuration rejected
Difficult final-test case
Solve 157x + 149y = 31 and 163x + 151y = -47.
Expected {"x":"-2921/145","y":"3108/145"}
Actual output
✓
{"x":"-2921/145","y":"3108/145"}
This case passed
Cheapest passing cost per 1,000 production calls
Waiting for a pass
- The replay orders eight settled model, reasoning, and prompt configurations from highest measured production cost to lowest.
- The frozen suite contains 30 exact-arithmetic systems; six distinct final-test systems are each run twice.
- Evalt tries a Cramer's-rule rewrite, which improves one DeepSeek configuration from 50 to 66.7 percent but remains below the gate.
- GPT OSS 20B at medium reasoning passes every repeated final-test execution at about $2.13 per thousand calls.
- Cheaper low-reasoning and small-model configurations fail, confirming the selected route is the cheapest measured configuration that works.
New production routeDefine the task once. Keep improving what serves it.
First answer
What happens after this?
We infer a draft success rule, ask for only the next examples that reduce uncertainty, reserve unseen examples, improve the prompt iteratively, and show the cheapest model that still clears your quality floor. One answer starts setup; it never counts as proof by itself.
Step 1 of 5
What should this prompt accomplish?
Start with the exact job, prompt, and model you need to improve.
Draft projectAI drafts, I approve$0.50 cap
- 1Job
- 2Cases
- 3Approve
- 4Run
- 5Winner
Local image suite
Pick an image. Write the answer you know is right.
No AI is needed to create ground truth. Images stay in this tab while you edit; Evalt writes them only into the suite you explicitly download.
What happens next: attach one PNG, JPEG, WebP, or GIF to each single-turn scenario, add an accessible description, and write the expected answer. Evalt downloads an evalt-suite-v2 file for offline validation and a bounded local run. Hosted jobs do not receive or retain these images.
What must never break?
Hard rules veto a candidate even when its average score rises.
What leaves this browser?
Nothing in this replay preview.
Run from your machine
Your suite is ready for an offline check.
The download contains the images as data URLs. Keep it private. Validation makes no provider call; optimization sends the approved cases only to the named image-capable OpenRouter routes under Evalt's enforced privacy policy and cost cap.
evalt validate image-evaluation-evalt.json && evalt optimize image-evaluation-evalt.json --output evalt-result.jsonOpen the frozen contract
Optimization frontier
Watch quality compete with cost.
Cheapest passing nowSearching…
Validation accuracy ↑Provider cost →
Cheapest passing cost over time
Live evidence
Waiting for the first response…See every answer and verdict as it lands.
Outputs, pass/fail decisions, judge reasons, prompt revisions, and model names will stream here.
Final test result
Cost-quality frontier
Original prompt
BaselineTested prompt
What still fails
Candidate-by-candidate evidence
Automatic route maintenance
Retest when the economics change.
Set a hard maintenance cap. Evalt queues a new tournament when traffic supplies enough fresh outcomes, a model appears, or OpenRouter prices change; a challenger is promoted only after it passes the frozen contract.
Use the result
Put the same quality gate in CI.
The exported report is machine-readable. Fail a build when the selected prompt drops below your floor.
evalt check evalt-result.json --min-pass-rate 0.95Measured only on this frozen replay fixture. This is product QA, not a claim about your prompt, demand, or future model versions.
Not the smartest model. The right model for this job.
Evalt rejects models that are too weak, then stops paying for extra intelligence once a cheaper route proves it can do the work reliably.
ConfigurationWeighted finalCost / 1KDecision
GPT-5.6 Luna · low83.3%$49.25Below gate
MiniMax M3 · low100%$9.86Passes, costs more
DeepSeek V4 · high83.3%$5.91Below gate
DeepSeek V4 · rewrite + medium66.7%$2.98Rewrite still fails
GPT OSS 120B · medium100%$2.30Passes, costs more
GPT OSS 20B · medium100%$2.13Selected
GPT OSS 20B · low83.3%$1.07Too inaccurate
Ling 2.6 Flash · none0%$0.002Too inaccurate
Every route faces the same unseen final-test cases and the same 95% requirement.
Repeated executions expose a route that passes once and fails the next time.
Hard rules can veto an attractive average score.
Open the exact case, model output, and decision behind every score.
Live spend stays visible under a server-enforced cap.
Export the prompt, frozen contract, results, and model labels.
Start with a quality target. Take control when you need it.
Require target_accuracy=0.95; Evalt selects the lowest-cost tested configuration that reaches it.
Set incumbent_model with objective="match_baseline_at_lowest_cost" only when an existing route's measured quality should be preserved.
Set price_usd with objective="best_within_price" to maximize measured accuracy under a fixed per-call cap. It may not reach 95%.
test_budget_usd="auto" is capped by max_test_budget_usd; it is never permission for unlimited background spend.
Cost comparisons use the 90th-percentile input and output lengths observed on the route.
Low, medium, and high effort compete as distinct configurations when a model supports them.
“100%” means 100% on the approved test set—not a promise that unknown future inputs cannot fail.
Ready to test a real task?