Copy this once
Evalt will not show this credential again.
Workspace
Production routes
See what is serving, what it costs, and whether it still clears your test.
Workspace access
Create a private workspace.
No account is required. Create an owner key, then grant only the access each person or CI job needs. Prompts, inputs, outputs, cases, responses, and provider keys stay local.
Workspace ID checking…. The SDK must show this same ID.
OwnerFull workspace control
python3 -m evalt connectTeam access
20 active capabilities maximumGive the minimum access needed
RouteQualityCost / 1KServingLast checked
No account requiredCreate a private key to sync sanitized route metadata.
Recorded example
security-answers
Answer customer security questions from approved company evidence without inventing claims.
Selected configurationGPT OSS 20B · medium
Observed final test12 / 12Quality floor95%Production cost$2.13 / 1KPromptStarting prompt
Selected only after the complete batch settled. A cheaper configuration was tested and failed the frozen quality gate.
Cost frontier
95% requiredWhy this route is serving
Higher production costLower production cost
Frozen contract v1
Human-reviewed structureEvidence before averages
Every candidate saw the same 30 systems. Six separate final-test systems were run twice.
Decision history
Every route change leaves a receipt.
Promotions, rejections, price changes, and manual approvals stay attached to this task.
GPT OSS 20B · medium promoted
100% observed final test · $2.13 / 1K calls · starting prompt retained
GPT OSS 20B · low rejected
Cheaper, but 83.3% missed the frozen 95% gate.
Prompt rewrite tested
Improved one DeepSeek lane from 50% to 66.7%; not promoted.
Contract v1 frozen
30 cases · repeated final test · $0.18 test cap.
Production handoff
Your code names the task, not today’s model.
The SDK executes the promoted configuration and keeps feedback isolated to this route.
Python
from evalt import Evalt
evalt = Evalt()
answer = evalt.run(
security_prompt,
customer_question,
route="security-answers",
target_accuracy=0.95,
)
print(answer.content)The selected prompt, model, reasoning effort, and price snapshot are versioned together.
Accepted and corrected outputs become evidence only for security-answers.
A challenger cannot replace the serving route unless it clears the same frozen contract.
Local route
route
Connected
Serving decisionWaiting for a tested routeNo reliability evidence yet
Serving baseline—Required quality—Test spend—Production calls0
Progress appears here while the local tournament runs. Dashboard downtime never blocks the SDK.
Decision evidence
WaitingWhy this configuration is serving
Route health
Not checkedRecheck the frozen winner
Run the selected prompt, model, few-shot package, evaluator, and final-test cases again. This never searches or changes the serving route.
Signed alertsOptional · configured and sent from your machineOff
Send only the aggregate verdict and deltas to your incident or automation system. The destination and signing secret never enter the suite, result, or dashboard.
$env:EVALT_WEBHOOK_SECRET="…" in PowerShell or export EVALT_WEBHOOK_SECRET="…" in a POSIX shell. Never paste the secret into this dashboard.
@name from evalt library. Full cases and outputs stay local; only the verdict and aggregate deltas can appear here.
Retained checks
Serving baseline to latest
Inspect the exact cases locally
The dashboard has aggregates only. Render the latest saved result without a provider call.
Quality evidence
Does this route clear the bar?
Optimization funnel
What reached a decision
Configurations compared
0 testedQuality is shown against one required floor. Skipped configurations were not scored.
Detailed costs and reasons 0 configurations
Recent runs
Up to 12 privacy-safe summariesQuality over retained history
Run comparison
Aggregate metadata onlyWhat changed since the baseline?
Run this route twice to compare measured decisions.
Need promotion-grade evidence?
Compare saved case-level results on the connected computer.
python3 -m evalt compare baseline.json candidate.json --html comparison.htmlRaw diagnostics 0 events
Activity
Evidence status
No synchronized route decisions yet
Run Evalt locally or synchronize existing routes to populate this dashboard.
Open source on GitHub
Hit your quality bar. Cut the AI cost.
Give Evalt a recurring AI task and a quality target. It tests prompts, models, and reasoning levels on one frozen contract, then serves the lowest-cost configuration that passes.
Quality target 95%Tested 1 of 8Cheapest pass Searching
GPT-5.6 Luna · low reasoning
Overall: 5/6 systems · 83.3% · configuration rejected
Difficult final-test case
Solve 157x + 149y = 31 and 163x + 151y = -47.
Expected {"x":"-2921/145","y":"3108/145"}
Actual output
✓
{"x":"-2921/145","y":"3108/145"}
This case passed
Cheapest passing cost per 1,000 production calls
Waiting for a pass
- The replay orders eight settled model, reasoning, and prompt configurations from highest measured production cost to lowest.
- The frozen suite contains 30 exact-arithmetic systems; six distinct final-test systems are each run twice.
- Evalt tries a Cramer's-rule rewrite, which improves one DeepSeek configuration from 50 to 66.7 percent but remains below the gate.
- GPT OSS 20B at medium reasoning passes every repeated final-test execution at about $2.13 per thousand calls.
- Cheaper low-reasoning and small-model configurations fail, confirming the selected route is the cheapest measured configuration that works.
New production routeDefine the task once. Keep improving what serves it.
First answer
What happens after this?
We infer a draft success rule, ask for only the next examples that reduce uncertainty, reserve unseen examples, improve the prompt iteratively, and show the cheapest model that still clears your quality floor. One answer starts setup; it never counts as proof by itself.
Step 1 of 5
What should this prompt accomplish?
Start with the exact job, prompt, and model you need to improve.
Draft projectAI drafts, I approve$0.50 cap
- 1Job
- 2Cases
- 3Approve
- 4Run
- 5Winner
Local image suite
Pick an image. Write the answer you know is right.
No AI is needed to create ground truth. Images stay in this tab while you edit; Evalt writes them only into the suite you explicitly download.
What happens next: attach one PNG, JPEG, WebP, or GIF to each single-turn scenario, add an accessible description, and write the expected answer. Evalt downloads an evalt-suite-v2 file for offline validation and a bounded local run. Hosted jobs do not receive or retain these images.
What must never break?
Hard rules veto a candidate even when its average score rises.
What leaves this browser?
Nothing in this replay preview.
Run from your machine
Your suite is ready for an offline check.
The download contains the images as data URLs. Keep it private. Validation makes no provider call; optimization sends the approved cases only to the named image-capable OpenRouter routes under Evalt's enforced privacy policy and cost cap.
evalt validate image-evaluation-evalt.json && evalt optimize image-evaluation-evalt.json --output evalt-result.jsonOpen the frozen contract
Optimization frontier
Watch quality compete with cost.
Cheapest passing nowSearching…
Validation accuracy ↑Provider cost →
Cheapest passing cost over time
Live evidence
Waiting for the first response…See every answer and verdict as it lands.
Outputs, pass/fail decisions, judge reasons, prompt revisions, and model names will stream here.
Final test result
Cost-quality frontier
Original prompt
BaselineTested prompt
What still fails
Candidate-by-candidate evidence
Automatic route maintenance
Retest when the economics change.
Set a hard maintenance cap. Evalt queues a new tournament when traffic supplies enough fresh outcomes, a model appears, or OpenRouter prices change; a challenger is promoted only after it passes the frozen contract.
Use the result
Put the same quality gate in CI.
The exported report is machine-readable. Fail a build when the selected prompt drops below your floor.
evalt check evalt-result.json --min-pass-rate 0.95Measured only on this frozen replay fixture. This is product QA, not a claim about your prompt, demand, or future model versions.
The cheapest route that cleared 95%.
GPT OSS 20B at medium reasoning passed all 12 repeated final-test executions. The next cheaper configuration missed the floor; the next passing configuration cost more.
ConfigurationFinal testCost / 1KDecision
MiniMax M3 · low12 / 12$9.86Passes, costs more
GPT OSS 20B · medium12 / 12$2.13Selected
GPT OSS 20B · low11 / 12$1.07Below 95%
Ling 2.6 Flash0 / 12$0.002Too inaccurate
Start with a quality target. Take control when you need it.
Require target_accuracy=0.95; Evalt selects the lowest-cost tested configuration that reaches it.
Set incumbent_model with objective="match_baseline_at_lowest_cost" only when an existing route's measured quality should be preserved.
Set price_usd with objective="best_within_price" to maximize measured accuracy under a fixed per-call cap. It may not reach 95%.
test_budget_usd="auto" is capped by max_test_budget_usd; it is never permission for unlimited background spend.
Cost comparisons use the 90th-percentile input and output lengths observed on the route.
Low, medium, and high effort compete as distinct configurations when a model supports them.
“100%” means 100% on the approved test set—not a promise that unknown future inputs cannot fail.
Stable SDK · PyPI
Run the same decision on one real route.
evaltPyPIPython 3.10+
pip install evalt