Python SDK reference.

Run one durable route

Set OPENROUTER_API_KEY in your environment or a .env file in the directory where you run Python. Evalt() finds it automatically; an explicit api_key= takes precedence. The key is never written to Evalt’s route database or result files.

from evalt import Evalt

ticket = "Please help—the website won't load."
evalt = Evalt()  # interactive terminals show design, tournament, route, and cost

answer = evalt.run(
    "Classify this request. Return exactly one lowercase label: billing, account, or technical.",
    ticket,
    task="Route recurring support tickets to billing, account, or technical.",
    route="support-routing",
    target_accuracy=0.95,
    test_budget_usd="auto",
)

print(answer.content)

Connect local routes to the hosted workspace

The SDK remains the production runner. The optional workspace gives its routes a visual history and live progress view without moving customer content into the dashboard.

python3 -m evalt connect

The command creates one private workspace token in your user-level Evalt config, opens evalt.dev/app, and publishes existing sanitized route summaries from the current .evalt/evalt.db without making a model call. That connection follows scripts across project folders; pass --state path/to/evalt.db only when one project needs a separate workspace. The CLI and browser show a safe ws_... ID; those IDs must match. Future Evalt.run(...) calls publish bounded route progress and final operational summaries, and the terminal explicitly reports whether each sync succeeded. Every invocation has an opaque run ID and an explicit running, completed, or failed state, so two tournaments on the same route never blend together. Prompts, inputs, outputs, cases, provider keys, request bodies, and raw responses stay local. Dashboard outages never fail a production call.

The first evw_... capability is the owner key. Keep it out of CI and shared channels. Open Workspace access in the dashboard to create expiring evc_... Viewer, Publisher, or Editor capabilities. The CLI reports the effective role and permissions without printing the capability.

Evalt · support-routing · TEST DESIGN STARTED · 25 cases · current catalog-selected designer · deadline 120s · one workflow cap $1.00
Evalt · support-routing · DESIGNING TESTS · openai/gpt-5.4-mini · 10s elapsed · still working
Evalt · support-routing · TEST DRAFT READY · 25 AI-authored cases · $0.94 remains for the tournament
Evalt · support-routing · BROAD SCREEN · 8 model configuration(s) · up to 8 in parallel
Evalt · support-routing · SCREENED · qwen/qwen3.5-9b#reasoning=low · 100% validation · p90 920 ms · $0.041200 spent
Evalt · support-routing · ROUTE SELECTED · qwen/qwen3.5-9b#reasoning=low · 100% observed final test · $0.31 test spend
Evalt · support-routing · PROVISIONAL EVIDENCE · 74.1% one-sided 95% lower bound · 10 distinct scenarios

Inspect the AI-built test before it runs.

Evalt selects a capable test designer from the current intelligence-and-price catalog, then drafts 25 varied routine, ambiguous, adversarial, boundary, and realistic-domain scenarios in five parallel batches before the split. It recommends exact, semantic, or explicit numeric-tolerance judging. A 0–100 subjective rating can therefore accept a nearby approved score without asking another model to pretend one arbitrary number is uniquely correct. The designer fallback chain is a separate live-catalog shortlist and never borrows the judge. The draft spends from the same workflow cap as the later tournament. Evalt names the active model and attempt; if a provider returns malformed structured output, it rejects that draft and makes one visible budget-bounded recovery attempt before falling back.

from evalt import Evalt

evalt = Evalt()
draft = evalt.optimize_task(
    task="Route recurring support tickets to billing, account, or technical.",
    prompt="Return exactly one lowercase label: billing, account, or technical.",
    route="support-routing",
    case_control="review",
    workflow_budget_usd=1.00,
)

draft.save("support-routing-draft.json")
for case in draft.examples:
    print(case.id, case.conversation())
if input("Type APPROVE after reviewing every expected output: ").strip() != "APPROVE":
    raise SystemExit("Draft saved; no tournament ran.")

# Explicit trust boundary; pass edited examples to approve(...) when needed.
suite = draft.approve()
result = evalt.run(suite)

After approval, the tournament tries current model candidates and supported reasoning levels in parallel. Prompt rewrites and approved training-only few-shot packages compete too. With the default balanced 25-case suite, ten cases train prompt packages, five choose among them, and ten unique unseen cases confirm the frozen winner. Each final case runs twice, for twenty final executions; prompt tuning never sees them.

Route lifecycle

A route is one stable production task with its own prompt, examples, budget, selected configuration, and maintenance history.

status = evalt.route_status("support-routing")

print(status["selected_model"])
print(status["current_package_id"])
print(status["selected_prompt_version"])
print(status["target_accuracy"])
print(status["test_budget_usd"])
print(status["maintenance_due"])

support = evalt.run(support_prompt, ticket, route="support-routing")
summary = evalt.run(summary_prompt, transcript, route="call-summary")
fraud = evalt.run(risk_prompt, transaction, route="fraud-review")

Each route keeps separate prompts, evidence, winners, budgets, and retest history. Feedback never crosses route names.

Test the request you will actually serve.

Put Chat Completions settings in request_options. Evalt sends the same canonical envelope to every target-model training, validation, and final-test call, stores its SHA-256 fingerprint with the winning route, and reuses it when later calls omit the argument. Test design, prompt optimization, and judging keep independent settings.

answer = evalt.run(
    prompt,
    ticket,
    route="support-routing",
    max_tokens=2048,
    request_options={
        "temperature": 0.2,
        "top_p": 0.95,
        "response_format": {
            "type": "json_schema",
            "json_schema": {
                "name": "support_route",
                "strict": True,
                "schema": route_schema,
            },
        },
        "tools": tools,
        "tool_choice": "auto",
        "parallel_tool_calls": False,
        "provider": {
            "require_parameters": True,
            "data_collection": "deny",
            "zdr": True,
        },
        "plugins": [{"id": "response-healing"}],
    },
)

print(answer.content)
print(answer.tool_calls)
print(answer.request_envelope_validated)

Compare models on images you actually approved.

Evalt accepts local PNG, JPEG, WebP, and GIF files or public HTTPS image URLs whose path ends in one of those extensions through the Chat Completions image-understanding path. It filters out text-only candidates before spend, keeps the image modality in the tested request contract, and never sends raw images, filenames, paths, source URLs, or thumbnails to the hosted dashboard.

You do not need AI to make the examples: choose images whose answers you already know, write the expected label yourself, and review the set before running it. AI-generated or automatically inferred image labels do not count as approved ground truth.

evalt import-dataset package-labels.csv \
  --prompt "Inspect the package. Return exactly one label: damaged or intact." \
  --input-text "Inspect this image." \
  --image-field image_path \
  --expected-field label \
  --id-field case_id \
  --group-field condition \
  --evaluator exact_text \
  --model google/gemini-2.5-flash-lite \
  --model openai/gpt-5.4-nano \
  --confirm-expected-human-approved \
  --output package-condition.json

The importer reads CSV, JSONL, or a JSON array entirely offline. Mappings are exact; Evalt does not guess the input, expected answer, ID, group, weight, or image columns. Use --input-field text instead of --input-text when text differs by row. Image paths must be relative to the dataset file. The importer validates local PNG, JPEG, and WebP files, rejects URLs and directory traversal, refuses ambiguous mappings, duplicate IDs, malformed records, unsafe sizes, and existing outputs, then emits a deterministic evalt-suite-v2 file. Its machine-readable summary contains only counts and a hash—never prompts, rows, labels, paths, or images—and it does not read provider or dashboard credentials.

from evalt import Evalt, Example, ImageInput, Suite, multimodal_input

examples = tuple(
    Example(
        multimodal_input("Return exactly one label: damaged or intact.", ImageInput.from_path(path)),
        approved_output=label,
        id=f"package-{index}",
    )
    for index, (path, label) in enumerate(approved_images, start=1)
)

suite = Suite(
    name="package-condition",
    prompt="Inspect the package image. Return exactly one label: damaged or intact.",
    examples=examples,
    models=("google/gemini-2.5-flash-lite", "openai/gpt-5.4-nano"),
    evaluator={"type": "exact_text"},
    optimize_prompt=False,
    max_optimization_cost_usd=0.25,
)

result = Evalt().run(suite)
print(result.winner.model, result.winner.holdout_pass_rate)

Know which frozen cases produced every decision.

Turn an approved suite into an immutable local history. Each revision names its exact parent, requires stable case IDs, and records whether cases were added, removed, changed, or left untouched. A stale base fails instead of silently replacing another reviewer's work.

evalt dataset create support support-v1.json \
  --message "Initial human-reviewed cases" \
  --tag reviewed

evalt dataset revise support support-v2.json \
  --base dv_0123456789abcdef0123456789abcdef \
  --message "Correct labels and add refund edge cases"

evalt dataset diff support \
  --from-version dv_0123456789abcdef0123456789abcdef

evalt dataset export support \
  --output support-production.json

See where the judge is wrong before trusting its score.

Create a private queue from one frozen suite and one matched set of judge decisions. Reviewers claim explicit cases, record a human pass or fail, and can skip or reopen an item without losing the queue's event history. Calibration then separates true passes, false passes, true failures, and false failures instead of hiding disagreement inside one average.

evalt review create support support-production.json \
  --decisions judge-decisions.json \
  --rubric "Pass only when the approved behavior is fully satisfied."

evalt review claim support case-17 \
  --reviewer reviewer-a \
  --revision qr_1_0123456789abcdef

evalt review resolve support case-17 \
  --reviewer reviewer-a \
  --human-fail \
  --corrected-output-file corrected.txt \
  --revision qr_2_0123456789abcdef

evalt review calibrate support \
  --output judge-calibration.json

false_pass_rate is the share of judge passes rejected by a human. false_fail_rate is the share of judge failures accepted by a human. A calibration report describes only the reviewed queue revision; it does not prove that one reviewer is universal ground truth. Run python -m evalt.examples.review_calibration for a complete packaged offline example.

Test the function your users call—not only the model behind it.

Compare customer-owned Python functions, agents, retrieval code, or RAG pipelines against one frozen set of approved cases. Every system receives the same stable case identity and an isolated deepcopy-compatible input. Evalt records quality, failures, p50 and p90 latency, configured cost, and the lowest-cost system clearing the floor without making a provider call or uploading the cases.

from evalt import CallableSystem, SystemExample, evaluate_systems

cases = (
    SystemExample(
        {"query": "How long are refunds available?", "documents": documents},
        "30 days",
        id="refund-window",
        critical=True,
    ),
)

result = evaluate_systems(
    cases,
    (
        CallableSystem("baseline", baseline),
        CallableSystem("rag-v2", rag_v2, cost_per_call_usd=0.0003),
    ),
    evaluator={"type": "exact_text"},
    quality_threshold=1.0,
)

print(result.selected_system)
result.save("application-eval.json")

Result export is content-free by default: it stores case IDs, decisions, metrics, and hashes instead of inputs or outputs. Use include_content=True only for an intentional private local artifact. The adapters read no provider or dashboard credential and perform no synchronization. Run python -m evalt.examples.callable_rag for the in-process example or python -m evalt.examples.process_supervision for the supervised-process example.

The SDK serves. The workspace explains why.

The free workspace is organized around durable named routes, with the latest local invocation scoped inside each route. Open one to see whether the current run is active, completed, or failed; its progress list excludes older invocations while the serving configuration and route history remain durable.

Share the workspace without sharing the owner key

Workspace access remains accountless. An owner creates a named, expiring capability and Evalt shows its secret once. Access links keep the capability in the URL fragment, so it is not sent in the page request, server logs, or referrer. The server stores only a one-way capability digest plus the bounded label, role, creation, expiry, revocation, and last-used times.

Accept, correct, or leave unjudged

Feedback is explicit. Evalt does not silently infer that a delivered answer was correct. Accept an outcome only when it meets the real task; correct it with the expected behavior when it does not.

Freeze a suite before provider spend

Use a suite when you already have approved cases, need an auditable tournament, or want a CI artifact. Validation is offline; optimization is the only command below that calls a provider.

status = evalt.route_status("support-routing")
version = status["current_package_id"]

answer = evalt.run(
    prompt,
    input,
    route="support-routing",
    route_version=version,
)

Safe rollout: qualify locally, read the new current_package_id, deploy that pin, and deliberately replace it only after review. A stale, missing, cross-route, unqualified, or damaged pin fails before any provider call. Rollback creates a new current package ID, so deploy that new ID rather than silently reusing the old one.

{
  "schema": "evalt-suite-v2",
  "name": "support-routing",
  "prompt": "Return billing, account, or technical.",
  "examples": [
    {"id": "billing-1", "input": "Charged twice", "approved_output": "billing"},
    {"id": "account-1", "input": "Reset link expired", "approved_output": "account"},
    {"id": "technical-1", "input": "App freezes", "approved_output": "technical"}
  ],
  "evaluator": {"type": "exact_text"},
  "quality_threshold": 0.95,
  "max_optimization_cost_usd": 2.0,
  "rounds": 3,
  "max_parallel_models": 16,
  "max_parallel_scenarios": 32,
  "request_timeout_seconds": 600
}

Model the bell curve without hiding the hard tail

An overall 95% can be misleading when routine requests dominate traffic. Use group to stratify related cases across training, validation, and final test; weight to approximate production frequency; and difficulty_thresholds to require separate routine, complex, and adversarial floors.

{
  "examples": [
    {"id":"routine-01", "group":"ordinary-refund", "difficulty":"routine", "weight":6, "input":"...", "approved_output":"approve"},
    {"id":"hard-01", "group":"policy-conflict", "difficulty":"adversarial", "weight":1, "critical":true, "input":"...", "approved_output":"manual_review"}
  ],
  "quality_threshold": 0.95,
  "difficulty_thresholds": {
    "routine": 0.95,
    "complex": 0.90,
    "adversarial": 0.85
  }
}

This is an excerpt. When grouping is enabled, every case needs a group and every group needs at least five cases. Promotion requires the overall weighted target and every named difficulty floor. Put rare catastrophic rules in a deterministic veto too.

Use the cheapest valid evaluator

Prefer deterministic evaluators whenever the contract allows it. Semantic judging is useful for nuanced outputs, but it adds provider cost and must be calibrated against route-specific human decisions before promotion.

from evalt import CommandScorer, Evalt, Suite
import sys

suite = Suite.load("evalt.json")
scorer = CommandScorer(
    "domain-rubric", "1.0",
    [sys.executable, "tools/score.py"],
    timeout_seconds=5,
)
result = Evalt(
    custom_scorers={scorer.scorer_id: scorer}
).run(suite)

A suite can contain only type, scorer_id, and scorer_version for a custom evaluator. It cannot choose a command, module, executable, or path. Command scorers use strict JSON stdin/stdout, no shell, bounded I/O, a per-case timeout, and a minimal environment that excludes provider credentials by default. Missing registration, version drift, timeout, nonzero exit, malformed output, and non-finite scores stop the run.

Keep a conversation in one split

A multi-turn scenario uses a turns array. Evalt keeps the full conversation together when creating training, validation, and final-test splits, and replays prior assistant context in order.

{
  "id": "refund-followup",
  "turns": [
    {"role": "user", "content": "I was charged after cancelling."},
    {"role": "assistant", "content": "When did you cancel?"},
    {"role": "user", "content": "Three days before renewal."}
  ],
  "approved_output": "billing"
}

Turns inside one scenario remain ordered. Independent scenarios and repeated executions may run concurrently.

One budget covers the whole search

max_optimization_cost_usd covers prompt improvement, candidate calls, judging, retries, and final testing. Evalt reserves estimated in-flight cost before dispatch; work that cannot fit does not start.

Raise concurrency to reduce wall time, not to expand spend. Model lanes are configurable up to 32 and independent scenario executions up to 128; every request still competes for the same reserved budget. The terminal reports the broad parallel screen, then prints each settled configuration's validation rate, p90 latency, spend, and total elapsed time. Automatic first-route tests generate five coverage batches concurrently and use a 120-second deadline for both designer and candidate requests. Set designer_request_timeout_seconds or test_request_timeout_seconds when the job needs different limits. A candidate that times out cannot earn a higher reasoning rung. Automatic routes try one prompt rewrite by default; set optimization_rounds from 1 to 8 for a deeper search. Explicit suites keep a configurable 600-second default and can raise request_timeout_seconds as high as 7200 seconds.

Models, prompts, and reasoning are configurations

Evalt begins across a current price/intelligence frontier, then spends the remaining budget near the observed task-capability band. Ordinary supported reasoning levels may run in the same parallel wave. Extra-high and maximum effort are staged: the prior rung must complete, remain within one validation case of the quality target, and satisfy the route's p90 latency ceiling. A timeout blocks escalation, and final-test luck never unlocks a higher rung. Reasoning-only hone lanes reuse the same frozen prompt and few-shot package so the comparison isolates effort.

• Catalog benchmarks shortlist candidates; they never promote a production route.
• Prompt revisions that improve validation propagate to screened models before those models can be pruned; propagated winners still face the untouched final test.
• Set optimize_prompt: false to hold the supplied prompt exactly fixed and compare only model, reasoning, and provider configurations.
• Few-shot examples come only from customer-approved training cases and are removed when evaluating their own scenario. The result records every demonstration ID with source_split: "train".
• A configuration must clear the approved final test—repeated when configured—before promotion.
• Production cost uses observed 90th-percentile input and output lengths instead of toy requests.

Run the same performance gate on every trusted change

The versioned Evalt Action can run a budget-capped tournament or gate an existing result without provider calls. It validates the suite before spending, writes JSON, HTML, and JUnit evidence, and exposes stable outputs for later deployment steps.

name: Evalt

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  performance-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: Find the cheapest route that passes
        id: evalt
        uses: JarJarBeatyourattitude/evalt-action@v1
        env:
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
        with:
          suite: evalt.json
          min-pass-rate: "0.95"
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalt-evidence
          path: |
            evalt-result.json
            evalt-report.html
            evalt-junit.xml

Store the OpenRouter key as a repository Actions secret. Ordinary secrets are not passed to workflows from forks; do not switch untrusted pull requests to pull_request_target merely to expose a provider key. See the Action repository for every input and output.

Use stable names instead of fragile file paths

The local evidence library gives approved suites and exported results immutable names such as @support-v1 and @support-baseline-v1. Evalt validates every import, preserves the original UTF-8 JSON bytes, verifies the SHA-256 digest on every read, and keeps the catalog on this computer. Library names, tags, paths, hashes, suites, results, prompts, images, cases, and outputs are never synchronized to the hosted workspace.

evalt library add evalt.json \
  --name support-v1 \
  --tag production

evalt library add evalt-result.json \
  --name support-baseline-v1 \
  --tag baseline

evalt validate @support-v1
evalt check @support-baseline-v1 --min-pass-rate 0.95
evalt monitor @support-baseline-v1 \
  --route support-routing \
  --max-cost-usd 0.10

@name works anywhere the CLI reads a suite or result: validate, optimize, monitor, report, compare, and check. For a non-default catalog, add --library-root path to those commands or set EVALT_LIBRARY_HOME. The default is .evalt/library in the current project.

For image routes, import the original reviewed suite too, then run evalt monitor @support-baseline-v1 --suite @support-images-v1 .... The library stores the exact local suite; the dashboard still receives only the aggregate health verdict and deltas.

Detect drift without changing the route

evalt monitor reruns only the selected prompt, model, few-shot package, evaluator, request options, final-test cases, and repeat count from an earlier result. It does not search, rewrite, promote, roll back, maintain, or otherwise change the serving route. A positive provider-spend cap is mandatory.

python3 -m evalt monitor @support-baseline-v1 \
  --route support-routing \
  --max-cost-usd 0.10 \
  --max-regressions 0 \
  --max-quality-drop-pp 0 \
  --max-cost-increase-pct 15 \
  --max-p90-increase-ms 250 \
  --output evalt-monitor-result.json \
  --history .evalt/monitor-history.jsonl

The full evalt-monitor-result-v1 file stays local and remains compatible with evalt check, evalt compare, HTML reports, and JUnit reports. Optional history is aggregate-only: timestamp, verdict, opaque suite hash, model, quality/cost/latency deltas, regression counts, and check spend. It never contains prompts, inputs, images, approved answers, model outputs, judge reasons, generation IDs, request bodies, or credentials. With --route and an existing workspace connection, only that aggregate verdict and those deltas synchronize to the route dashboard.

Opt-in scheduled notification

A scheduled GitHub workflow is the notification boundary: REGRESSION or ERROR fails the job and GitHub applies the repository's normal Actions notifications. The schedule is never enabled by Evalt, and every run consumes provider budget only up to the cap shown in the workflow.

name: Evalt route health

on:
  workflow_dispatch:
  schedule:
    - cron: "17 8 * * 1"

permissions:
  contents: read

jobs:
  frozen-recheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: python3 -m pip install --upgrade -r https://evalt.onrender.com/python-sdk/latest.txt
      - name: Recheck the frozen winner
        env:
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
        run: |
          python3 -m evalt monitor evalt-baseline.json \
            --route support-routing \
            --max-cost-usd 0.10 \
            --max-regressions 0 \
            --max-quality-drop-pp 0 \
            --output evalt-monitor-result.json
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: evalt-route-health
          path: evalt-monitor-result.json

Run schedules only on a trusted branch. Commit a baseline only when its approved text cases are repository-safe. Keep private baselines and image suites in an encrypted runner secret or on a controlled self-hosted runner. HEALTHY means no configured regression was observed; it does not turn a small or provisional test set into a universal reliability guarantee.

Alert your systems without exporting evaluation content

Evalt can send a versioned aggregate event after a frozen route-health check or GitHub Action gate. The default payload contains an opaque route reference, verdict, suite hash, quality delta, regression and missing-case counts, cost change, p90 change, and measured check spend. It never contains prompts, inputs, approved answers, images, outputs, judge reasons, scorer identity or code, provider keys, the destination URL, or the signing secret.

export EVALT_WEBHOOK_SECRET="replace-with-a-long-random-secret"

python3 -m evalt monitor @support-baseline-v1 \
  --route support-routing \
  --max-cost-usd 0.10 \
  --webhook-url https://alerts.example.com/evalt \
  --webhook-destination-id incident-pipeline \
  --webhook-secret-env EVALT_WEBHOOK_SECRET \
  --webhook-required \
  --webhook-audit .evalt/webhook-deliveries.jsonl

Delivery uses HTTPS only, rejects credentials and fragments in the URL, follows no redirects, resolves and pins a public address for each attempt, and denies private, loopback, link-local, reserved, multicast, and unspecified destinations by default. It retries only bounded transport failures and selected retryable HTTP statuses, honors a capped numeric Retry-After, and retains no response body.

The local JSONL audit contains the exact aggregate event, status codes, retry timing, opaque destination label, and delivery result. Replay preserves the original event and idempotency identity:

python3 -m evalt webhook replay evt_0123456789abcdef0123456789abcdef \
  --webhook-url https://alerts.example.com/evalt \
  --webhook-destination-id incident-pipeline \
  --webhook-secret-env EVALT_WEBHOOK_SECRET \
  --webhook-audit .evalt/webhook-deliveries.jsonl

The private-network override exists for explicitly trusted self-hosted environments. Enabling it weakens the default SSRF boundary. Route names are also omitted by default; add --webhook-include-route-name only when the destination is authorized to receive that label.

Make measured quality a deploy check

python3 -m evalt check candidate.json \
  --baseline baseline.json \
  --min-pass-rate 0.95 \
  --max-cost-per-success 0.002 \
  --max-regressions 0 \
  --max-quality-drop-pp 0 \
  --require-complete-coverage

python3 -m evalt check is offline. With --baseline, it first requires identical non-empty suite hashes, then rejects missing cases, newly failing frozen cases, or aggregate quality loss. Optional --max-cost-increase-pct and --max-p90-increase-ms limits add measured efficiency guardrails. It does not call OpenRouter or mutate either result.

Portable reports and run comparison

Render saved results or compare two frozen runs without provider access. A comparison is promotion-safe only when both results pin the same suite hash.

evalt report evalt-result.json \
  --html evalt-report.html \
  --junit evalt-report.xml

evalt compare baseline.json candidate.json \
  --output comparison.json \
  --html comparison.html

evalt compare is the detailed local diagnosis; evalt check --baseline is the deployment decision. Both use the same case-level comparison and refuse promotion when suite hashes differ or are absent.

Know what leaves your machine

• Every OpenRouter request asks for Zero Data Retention and denies provider data collection.
• Endpoint eligibility still depends on the current provider route; Evalt fails closed when the required privacy route is unavailable.
• Current pricing is refreshed at least hourly by default. An unpriced or unbounded route is ineligible.
• The open-source SDK adds no platform fee. Provider charges remain visible in the result.

Stable contracts, explicit deprecations

Evalt follows semantic versioning for the evalt Python API, CLI command names, suite schema, result schema, and exit-code contract. Deprecations remain for at least one minor release and are documented before removal.

The historical modelsieve, last_good_prompt, and lgp names are compatibility shims. New integrations should use the evalt import and command.

Common failure states