Recorded result · support classification
Same observed quality. A much cheaper passing route.
This is a real saved Evalt run, not a claim that one model is universally better. Every configuration below faced the same frozen billing/account/technical contract.
What was tested
The suite contains 30 approved JSON classification cases across billing, account, and technical requests, including ambiguous precedence cases. The frozen final test used six distinct scenarios, each executed twice. The quality floor was 95%; the selected route scored 100% in this recorded run.
Six final-test scenarios are enough to compare this run’s candidates, but not enough to prove a 95% population-level reliability claim. Evalt preserves that distinction instead of upgrading a perfect observation into statistical certainty.
Passing candidates were ranked by production cost
| Configuration | Observed final | Estimated cost / 1K | Decision |
|---|---|---|---|
| GPT-5 Mini · low | 100% | $0.2095 | Passes, costs more |
| Gemini 3 Flash Preview · none | 100% | $0.1940 | Passes, costs more |
| MiniMax M3 · none | 100% | $0.0577 | Passes, costs more |
| Qwen 3.5 9B · none | 100% | $0.0480 | Passes, costs more |
| GPT OSS 120B · low | 100% | $0.0188 | Passes, costs more |
| GPT OSS 20B · low | 100% | $0.0170 | Selected |
Relative to the passing GPT-5 Mini candidate in the same run, the selected estimated production cost was 91.9% lower. Total provider spend for the recorded tournament was $0.1715. Prices and model behavior can change; a production route should be monitored and retested.
What the selected route does
input: "I was charged twice for the same monthly plan."
output: {"route":"billing"}
input: "Resetting my password worked, but every project now times out."
output: {"route":"technical"}The winning prompt contains explicit precedence rules for mixed billing, account, and technical requests. Exact cases, outputs, prompts, and provider responses remain in the local result; the hosted dashboard receives aggregate metadata only.
How to evaluate Evalt safely
- Pin the SDK version and start with a noncritical route.
- Review or replace the evaluation cases with your own approved examples.
- Set an explicit provider-spend cap.
- Inspect the local JSON/HTML evidence and independently verify surprising results.
- Keep the route provisional until the evidence strength matches the production risk.