Evalt
Interactive replay demo · no data uploaded

Evidence-bound AI cost optimization

Keep the quality. Cut the AI cost.

Recorded optimization runSolve equations correctly in a fixed format
Quality floor 80%Attempt 1 of 6Cheapest pass Searching
Held-out case Solve 3(x + 4) = 27. Expected: {"x":5,"check":"3(5+4)=27"}
Qwen 3.5 9B Original prompt 25% · rejected below the quality floor
Prompt sent Solve for x.
Held-out result x = 9
Cheapest passing cost per 1K cases Waiting for a pass
Wrong answer: it ignores the parentheses.Testing six configurations
  1. Original Qwen prompt: 25 percent, rejected.
  2. JSON-only Gemini prompt: 50 percent, rejected because the required fields are missing.
  3. Exact-schema Mistral prompt: 75 percent, rejected because its substitution check is wrong.
  4. Verify-substitution DeepSeek prompt: 83 percent and 38 cents per 1,000 cases, first pass.
  5. Reason-and-verify GPT-5.6 prompt: 92 percent and 4 dollars 10 cents per 1,000 cases, passes but costs more.
  6. Propagated-verification Qwen Flash prompt: 83 percent and 11 cents per 1,000 cases, selected as the cheapest pass.

Point Evalt at the prompt and model you use now. It tests cheaper models, prompt revisions, few-shot examples, and reasoning levels—then replaces your route only when a lower-cost configuration matches or beats the quality your approved cases measured.

You give
Your current route and examples of what a correct answer looks like
Evalt proves
Which cheaper configuration matches the incumbent on held-out cases
It keeps
The savings, the regression gate, and an audit trail for every replacement
terminal
answer = evalt.run(prompt, input, route="support", incumbent_model=CURRENT_MODEL)
New evaluationChoose who controls the test
change it before the run
More AIEvaluation controlMore hands-on

Balanced setup: AI suggestions stay excluded until you review and approve them.

What happens after this?

We infer a draft success rule, ask for only the next examples that reduce uncertainty, reserve unseen examples, improve the prompt iteratively, and show the cheapest model that still clears your quality floor. One answer starts setup; it never counts as proof by itself.

The deliverable is evidence, not vibes

Six questions answered before you ship the rewrite.

The real Evalt optimization dashboard showing the original prompt, held-out verdict, cost frontier, and model comparison
The actual Evalt result screen.Open the full-size dashboard to inspect the tested prompt, held-out evidence, failed cases, and cost-quality frontier.
Did it improve?

Original and winner run on the same held-out cases.

Was it just lucky?

Repeated runs expose inconsistency in the target model.

What regressed?

Hard rules can veto an attractive average score.

What still fails?

Open the exact case, output, and failed requirement.

What did it cost?

Live spend stays visible under a server-enforced cap.

Can I prove it later?

Export the prompt, frozen contract, results, and model labels.

The production interface

Call Evalt instead of hard-coding a model.

Name the model you trust today. Evalt returns an answer immediately, learns from outcomes you accept or correct, and promotes a cheaper prompt/model/reasoning configuration only after it matches the incumbent on held-out cases. A separate capped test budget funds each comparison.

python -m pip install evalt-0.7.0-py3-none-any.whl

Download Python SDK
from evalt import Evalt

evalt = Evalt(api_key=OPENROUTER_API_KEY)
answer = evalt.run(
    prompt,
    ticket,
    route="support-routing",
    incumbent_model="openai/gpt-5-mini",
    test_budget_usd="auto",
)

send(answer.content)
answer.accept()          # or answer.correct(expected)

evalt.route_status("support-routing")

One default, advanced controls when you need them

The matched-quality default stays simple.

Default

incumbent_model establishes the quality to preserve; Evalt finds the lowest-cost configuration that matches it.

Accuracy target

Set target_accuracy with objective="lowest_cost_at_accuracy" when a fixed reliability gate is more useful than the incumbent.

Price-first mode

Set price_usd with objective="best_within_price" for a new workload with no trusted baseline.

Bounded testing

test_budget_usd="auto" is capped by max_test_budget_usd; it is never permission for unlimited background spend.

Real request sizes

Cost comparisons use the 90th-percentile input and output lengths observed on the route.

Reasoning is a candidate

Low, medium, and high effort compete as distinct configurations when a model supports them.

Accuracy is measured

“100%” means 100% on the approved held-out suite—not a promise that unknown future inputs cannot fail.

Use it your way

Start free. Pay for the part that keeps working.

The SDK is the adoption path. Pro is the business: encrypted history, CI gates, and automatic route maintenance after new models arrive. Managed inference is optional for teams that want one Evalt credential and one bill.

SDK + bring your own key

$0 platform fee

Run the durable Python router or this web workflow with your OpenRouter key. You pay the provider directly; Evalt adds no usage fee.

  • One-call runtime routing
  • SQLite decision and feedback history
  • Capped automatic model/prompt retests
Try with my key
Evalt Pro

$29 / month + usage

The subscription pays for continuous optimization. Use your own provider key, or fund optional managed inference at provider cost plus the current visible 20% service fee.

  • One Evalt credential and bill on the managed route
  • Every workflow keeps your hard spend cap
  • Raw prompts are never pooled for shared learning without explicit opt-in

The free demo and BYOK route are available without subscribing.

Secure checkout is processed by Stripe for Jonathan Larson, the operator of Evalt.