Python SDK reference.

Run one durable route

Set OPENROUTER_API_KEY in your environment or a .env file in the directory where you run Python. Evalt() finds it automatically; an explicit api_key= takes precedence. The key is never written to Evalt’s route database or result files.

from evalt import Evalt

ticket = "Please help—the website won't load."
evalt = Evalt()  # interactive terminals show design, tournament, route, and cost

answer = evalt.run(
    "Classify this request. Return exactly one lowercase label: billing, account, or technical.",
    ticket,
    task="Route recurring support tickets to billing, account, or technical.",
    route="support-routing",
    target_accuracy=0.95,
    test_budget_usd="auto",
)

print(answer.content)

Connect local routes to the hosted workspace

The SDK remains the production runner. The optional workspace gives its routes a visual history and live progress view without moving customer content into the dashboard.

python3 -m evalt connect

The command creates one private workspace token in your user-level Evalt config, opens evalt.dev/app, and publishes existing sanitized route summaries from the current .evalt/evalt.db without making a model call. That connection follows scripts across project folders; pass --state path/to/evalt.db only when one project needs a separate workspace. The CLI and browser show a safe ws_... ID; those IDs must match. Future Evalt.run(...) calls publish bounded route progress and final operational summaries, and the terminal explicitly reports whether each sync succeeded. Every invocation has an opaque run ID and an explicit running, completed, or failed state, so two tournaments on the same route never blend together. Prompts, inputs, outputs, cases, provider keys, request bodies, and raw responses stay local. Dashboard outages never fail a production call.

Evalt · support-routing · TEST DESIGN STARTED · 25 cases · current catalog-selected designer · deadline 120s · one workflow cap $1.00
Evalt · support-routing · DESIGNING TESTS · openai/gpt-5.4-mini · 10s elapsed · still working
Evalt · support-routing · TEST DRAFT READY · 25 AI-authored cases · $0.94 remains for the tournament
Evalt · support-routing · BROAD SCREEN · 8 model configuration(s) · up to 8 in parallel
Evalt · support-routing · SCREENED · qwen/qwen3.5-9b#reasoning=low · 100% validation · p90 920 ms · $0.041200 spent
Evalt · support-routing · ROUTE SELECTED · qwen/qwen3.5-9b#reasoning=low · 100% observed final test · $0.31 test spend
Evalt · support-routing · PROVISIONAL EVIDENCE · 74.1% one-sided 95% lower bound · 10 distinct scenarios

Inspect the AI-built test before it runs.

Evalt selects a capable test designer from the current intelligence-and-price catalog, then drafts 25 varied routine, ambiguous, adversarial, boundary, and realistic-domain scenarios in five parallel batches before the split. It recommends exact, semantic, or explicit numeric-tolerance judging. A 0–100 subjective rating can therefore accept a nearby approved score without asking another model to pretend one arbitrary number is uniquely correct. The designer fallback chain is a separate live-catalog shortlist and never borrows the judge. The draft spends from the same workflow cap as the later tournament. Evalt names the active model and attempt; if a provider returns malformed structured output, it rejects that draft and makes one visible budget-bounded recovery attempt before falling back.

from evalt import Evalt

evalt = Evalt()
draft = evalt.optimize_task(
    task="Route recurring support tickets to billing, account, or technical.",
    prompt="Return exactly one lowercase label: billing, account, or technical.",
    route="support-routing",
    case_control="review",
    workflow_budget_usd=1.00,
)

draft.save("support-routing-draft.json")
for case in draft.examples:
    print(case.id, case.conversation())
if input("Type APPROVE after reviewing every expected output: ").strip() != "APPROVE":
    raise SystemExit("Draft saved; no tournament ran.")

# Explicit trust boundary; pass edited examples to approve(...) when needed.
suite = draft.approve()
result = evalt.run(suite)

After approval, the tournament tries current model candidates and supported reasoning levels in parallel. Prompt rewrites and approved training-only few-shot packages compete too. With the default balanced 25-case suite, ten cases train prompt packages, five choose among them, and ten unique unseen cases confirm the frozen winner. Each final case runs twice, for twenty final executions; prompt tuning never sees them.

Route lifecycle

A route is one stable production task with its own prompt, examples, budget, selected configuration, and maintenance history.

status = evalt.route_status("support-routing")

print(status["selected_model"])
print(status["selected_prompt_version"])
print(status["target_accuracy"])
print(status["test_budget_usd"])
print(status["maintenance_due"])

support = evalt.run(support_prompt, ticket, route="support-routing")
summary = evalt.run(summary_prompt, transcript, route="call-summary")
fraud = evalt.run(risk_prompt, transaction, route="fraud-review")

Each route keeps separate prompts, evidence, winners, budgets, and retest history. Feedback never crosses route names.

Test the request you will actually serve.

Put Chat Completions settings in request_options. Evalt sends the same canonical envelope to every target-model training, validation, and final-test call, stores its SHA-256 fingerprint with the winning route, and reuses it when later calls omit the argument. Test design, prompt optimization, and judging keep independent settings.

answer = evalt.run(
    prompt,
    ticket,
    route="support-routing",
    max_tokens=2048,
    request_options={
        "temperature": 0.2,
        "top_p": 0.95,
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "support_route",
                "strict": True,
                "schema": route_schema,
            },
        },
        "tools": tools,
        "tool_choice": "auto",
        "parallel_tool_calls": False,
        "provider": {
            "require_parameters": True,
            "data_collection": "deny",
            "zdr": True,
        },
        "plugins": [{"id": "response-healing"}],
    },
)

print(answer.content)
print(answer.tool_calls)
print(answer.request_envelope_validated)

Compare models on images you actually approved.

Evalt accepts local PNG, JPEG, WebP, and GIF files or public HTTPS image URLs whose path ends in one of those extensions through the Chat Completions image-understanding path. It filters out text-only candidates before spend, keeps the image modality in the tested request contract, and never sends raw images, filenames, paths, source URLs, or thumbnails to the hosted dashboard.

You do not need AI to make the examples: choose images whose answers you already know, write the expected label yourself, and review the set before running it. AI-generated or automatically inferred image labels do not count as approved ground truth.

from evalt import Evalt, Example, ImageInput, Suite, multimodal_input

examples = tuple(
    Example(
        multimodal_input("Return exactly one label: damaged or intact.", ImageInput.from_path(path)),
        approved_output=label,
        id=f"package-{index}",
    )
    for index, (path, label) in enumerate(approved_images, start=1)
)

suite = Suite(
    name="package-condition",
    prompt="Inspect the package image. Return exactly one label: damaged or intact.",
    examples=examples,
    models=("google/gemini-2.5-flash-lite", "openai/gpt-5.4-nano"),
    evaluator={"type": "exact_text"},
    optimize_prompt=False,
    max_optimization_cost_usd=0.25,
)

result = Evalt().run(suite)
print(result.winner.model, result.winner.holdout_pass_rate)

The SDK serves. The workspace explains why.

The free workspace is organized around durable named routes, with the latest local invocation scoped inside each route. Open one to see whether the current run is active, completed, or failed; its progress list excludes older invocations while the serving configuration and route history remain durable.

Accept, correct, or leave unjudged

Feedback is explicit. Evalt does not silently infer that a delivered answer was correct. Accept an outcome only when it meets the real task; correct it with the expected behavior when it does not.

Freeze a suite before provider spend

Use a suite when you already have approved cases, need an auditable tournament, or want a CI artifact. Validation is offline; optimization is the only command below that calls a provider.

{
  "schema": "evalt-suite-v2",
  "name": "support-routing",
  "prompt": "Return billing, account, or technical.",
  "examples": [
    {"id": "billing-1", "input": "Charged twice", "approved_output": "billing"},
    {"id": "account-1", "input": "Reset link expired", "approved_output": "account"},
    {"id": "technical-1", "input": "App freezes", "approved_output": "technical"}
  ],
  "evaluator": {"type": "exact_text"},
  "quality_threshold": 0.95,
  "max_optimization_cost_usd": 2.0,
  "rounds": 3,
  "max_parallel_models": 16,
  "max_parallel_scenarios": 32,
  "request_timeout_seconds": 600
}

Model the bell curve without hiding the hard tail

An overall 95% can be misleading when routine requests dominate traffic. Use group to stratify related cases across training, validation, and final test; weight to approximate production frequency; and difficulty_thresholds to require separate routine, complex, and adversarial floors.

{
  "examples": [
    {"id":"routine-01", "group":"ordinary-refund", "difficulty":"routine", "weight":6, "input":"...", "approved_output":"approve"},
    {"id":"hard-01", "group":"policy-conflict", "difficulty":"adversarial", "weight":1, "critical":true, "input":"...", "approved_output":"manual_review"}
  ],
  "quality_threshold": 0.95,
  "difficulty_thresholds": {
    "routine": 0.95,
    "complex": 0.90,
    "adversarial": 0.85
  }
}

This is an excerpt. When grouping is enabled, every case needs a group and every group needs at least five cases. Promotion requires the overall weighted target and every named difficulty floor. Put rare catastrophic rules in a deterministic veto too.

Use the cheapest valid evaluator

Prefer deterministic evaluators whenever the contract allows it. Semantic judging is useful for nuanced outputs, but it adds provider cost and must be calibrated against route-specific human decisions before promotion.

Keep a conversation in one split

A multi-turn scenario uses a turns array. Evalt keeps the full conversation together when creating training, validation, and final-test splits, and replays prior assistant context in order.

{
  "id": "refund-followup",
  "turns": [
    {"role": "user", "content": "I was charged after cancelling."},
    {"role": "assistant", "content": "When did you cancel?"},
    {"role": "user", "content": "Three days before renewal."}
  ],
  "approved_output": "billing"
}

Turns inside one scenario remain ordered. Independent scenarios and repeated executions may run concurrently.

One budget covers the whole search

max_optimization_cost_usd covers prompt improvement, candidate calls, judging, retries, and final testing. Evalt reserves estimated in-flight cost before dispatch; work that cannot fit does not start.

Raise concurrency to reduce wall time, not to expand spend. Model lanes are configurable up to 32 and independent scenario executions up to 128; every request still competes for the same reserved budget. The terminal reports the broad parallel screen, then prints each settled configuration's validation rate, p90 latency, spend, and total elapsed time. Automatic first-route tests generate five coverage batches concurrently and use a 120-second deadline for both designer and candidate requests. Set designer_request_timeout_seconds or test_request_timeout_seconds when the job needs different limits. A candidate that times out cannot earn a higher reasoning rung. Automatic routes try one prompt rewrite by default; set optimization_rounds from 1 to 8 for a deeper search. Explicit suites keep a configurable 600-second default and can raise request_timeout_seconds as high as 7200 seconds.

Models, prompts, and reasoning are configurations

Evalt begins across a current price/intelligence frontier, then spends the remaining budget near the observed task-capability band. Ordinary supported reasoning levels may run in the same parallel wave. Extra-high and maximum effort are staged: the prior rung must complete, remain within one validation case of the quality target, and satisfy the route's p90 latency ceiling. A timeout blocks escalation, and final-test luck never unlocks a higher rung. Reasoning-only hone lanes reuse the same frozen prompt and few-shot package so the comparison isolates effort.

• Catalog benchmarks shortlist candidates; they never promote a production route.
• Prompt revisions that improve validation propagate to screened models before those models can be pruned; propagated winners still face the untouched final test.
• Set optimize_prompt: false to hold the supplied prompt exactly fixed and compare only model, reasoning, and provider configurations.
• Few-shot examples come only from customer-approved training cases and are removed when evaluating their own scenario. The result records every demonstration ID with source_split: "train".
• A configuration must clear the approved final test—repeated when configured—before promotion.
• Production cost uses observed 90th-percentile input and output lengths instead of toy requests.

Run the same performance gate on every trusted change

The versioned Evalt Action can run a budget-capped tournament or gate an existing result without provider calls. It validates the suite before spending, writes JSON, HTML, and JUnit evidence, and exposes stable outputs for later deployment steps.

name: Evalt

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  performance-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: Find the cheapest route that passes
        id: evalt
        uses: JarJarBeatyourattitude/evalt-action@v1
        env:
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
        with:
          suite: evalt.json
          min-pass-rate: "0.95"
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalt-evidence
          path: |
            evalt-result.json
            evalt-report.html
            evalt-junit.xml

Store the OpenRouter key as a repository Actions secret. Ordinary secrets are not passed to workflows from forks; do not switch untrusted pull requests to pull_request_target merely to expose a provider key. See the Action repository for every input and output.

Make measured quality a deploy check

python3 -m evalt check evalt-result.json \
  --min-pass-rate 0.95 \
  --max-cost-per-success 0.002 \
  --require-complete-coverage

python3 -m evalt check is offline. It does not call OpenRouter or mutate the saved result.

Portable reports and run comparison

Render saved results or compare two frozen runs without provider access. A comparison is promotion-safe only when both results pin the same suite hash.

evalt report evalt-result.json \
  --html evalt-report.html \
  --junit evalt-report.xml

evalt compare baseline.json candidate.json \
  --output comparison.json \
  --html comparison.html

evalt compare shows case-level improvements and regressions plus quality, cost, latency, model, and prompt deltas. If suite hashes differ or are absent, the report labels the comparison descriptive and refuses to imply a shared promotion gate.

Know what leaves your machine

• Every OpenRouter request asks for Zero Data Retention and denies provider data collection.
• Endpoint eligibility still depends on the current provider route; Evalt fails closed when the required privacy route is unavailable.
• Current pricing is refreshed at least hourly by default. An unpriced or unbounded route is ineligible.
• The open-source SDK adds no platform fee. Provider charges remain visible in the result.

Stable contracts, explicit deprecations

Evalt follows semantic versioning for the evalt Python API, CLI command names, suite schema, result schema, and exit-code contract. Deprecations remain for at least one minor release and are documented before removal.

The historical modelsieve, last_good_prompt, and lgp names are compatibility shims. New integrations should use the evalt import and command.

Common failure states