> ## Documentation Index
> Fetch the complete documentation index at: https://openrouter.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Auto Exacto

> Automatic tool-calling provider optimization

Auto Exacto is a routing step that automatically optimizes provider ordering for all requests that include tools. It runs by default on every tool-calling request, requiring no configuration.

## How It Works

When your request includes tools, Auto Exacto reorders the available providers for your chosen model using a combination of real-world performance signals:

* **Throughput** -- real-time tokens-per-second metrics (visible on the [Performance tab](https://openrouter.ai/models) of any model page).
* **Tool-calling success rate** -- how reliably each provider completes tool calls (also visible on the Performance tab).
* **Benchmark data** -- results from OpenRouter's own benchmark harness, which continuously benchmarks eligible provider endpoints. See [What the Benchmark Harness Runs](#what-the-benchmark-harness-runs) for the exact benchmarks and parameters, and [Where to Find the Scores](#where-to-find-the-scores) for where the results are published.

Providers that underperform on these signals are deprioritized, while providers with strong track records are moved to the front of the list.

## How Tool-Calling Success Rate Is Measured

The tool-calling success rate signal is derived from the **Tool Call Error Rate** metric, which is also visible on the Performance tab of any model page. For each request that includes tools, OpenRouter inspects every tool call the model returned and validates it against the schemas the caller supplied.

### Validator

Tool call `arguments` are validated against the corresponding `tools[].function.parameters` schema using [`@cfworker/json-schema`](https://www.npmjs.com/package/@cfworker/json-schema), pinned to **JSON Schema Draft 7**:

```ts lines theme={null}
new Validator(parameters, '7')
```

Tools whose `parameters` schema is absent or fails to compile are treated as having no schema and are always considered valid, so the metric is conservative when caller-side schemas are malformed.

### Regex engine

`@cfworker/json-schema` delegates `pattern` and `patternProperties` to the runtime's built-in regex implementation. In OpenRouter's environment that is the native JavaScript `RegExp` (V8 / ECMA-262). There is no ECMA-262-conformance shim layered on top, so JavaScript regex semantics differ in some edge cases from the regex dialect specified by JSON Schema.

### Per-tool-call classification

Each tool call is bucketed into one of three error categories, or counted as valid:

* **`InvalidJson`** -- `JSON.parse(arguments)` throws.
* **`UnknownName`** -- `function.name` is not present in the request's `tools[]`.
* **`SchemaMismatch`** -- the validator returns `valid: false` against the resolved schema.

### Request-level aggregation

A request is flagged as errored if **any** of its tool calls falls into one of the three buckets above. The Tool Call Error Rate displayed per endpoint per day is then computed at the **request** level -- both the numerator and the denominator are counts of requests, not counts of individual tool calls:

```lines theme={null}
requests_with_tool_call_errors / requests_where_finish_reason_is_tool_calls
```

In other words: of all the requests where the model finished by emitting tool calls, what fraction had at least one tool call that hit one of the three error buckets. A request with five tool calls and one invalid call counts as one errored request, not one-out-of-five.

### Caveats

* Keywords introduced only in JSON Schema Draft 2019-09 or 2020-12 (for example `unevaluatedProperties`, `$dynamicRef`) are not enforced under Draft 7.
* JavaScript regex semantics differ from the ECMA-262 regex dialect formally referenced by JSON Schema, so `pattern` checks may behave differently than a strict JSON Schema implementation would.

## What the Benchmark Harness Runs

The benchmark signal comes from OpenRouter's in-house benchmark harness, which runs on a recurring schedule against every provider endpoint serving an enrolled model. Two benchmarks currently feed Auto Exacto routing:

* **GPQA Diamond** -- graduate-level multiple-choice science questions. The benchmark fixes temperature at `0.5` (matching the reference openbench configuration), shuffles answer options deterministically per question, and runs 10 epochs (repeated passes over the dataset) by default.
* **Tau2-Bench Airline** -- an agentic tool-calling benchmark where an LLM user-simulator drives multi-turn conversations against airline-domain tools, with a binary reward from final database-state, golden-action, and communication-information checks. The benchmark fixes temperature at `0`, and runs 1 epoch by default.

### How runs are executed

* Each benchmark run **pins a specific provider endpoint**, so every score is attributable to exactly one endpoint with no fallback. An additional unpinned run against OpenRouter's default routing is executed as a baseline.
* Tool-requiring presets are skipped on endpoints without tool support.
* Scores used for routing are aggregated over a **rolling 32-day window**.
* Runs must meet a minimum sample-size floor to count (currently at least 50 GPQA questions and 45 Tau2 tasks), so partial or failed runs do not affect routing.

GPQA option shuffling deliberately differs from openbench: openbench reseeds per record so the correct answer is always `B`, while this harness shuffles by index, so its GPQA numbers are not directly comparable to openbench-published scores.

## Where to Find the Scores

Benchmark results are public:

* **Model pages** -- the **AutoExacto Benchmarks** card on each enrolled model's page shows per-provider (and, where providers run multiple endpoints, per-endpoint) scores over the same rolling window used for routing.
* **Performance tab** -- throughput and tool-call error rate for each endpoint are on the Performance tab of every model page.

## Results

We have observed notable improvements in [tau-bench](https://github.com/sierra-research/tau-bench) scores and tool-calling success rates when Auto Exacto is active.

## Opting Out

Without Auto Exacto, OpenRouter's default routing is primarily [price-weighted](/docs/guides/routing/provider-selection#price-based-load-balancing-default-strategy) -- requests are load balanced across providers with a strong preference for lower cost. Auto Exacto changes this for tool-calling requests by reordering providers based on quality signals instead of price.

If you want to restore the previous price-weighted behavior for tool-calling requests, you can opt out by explicitly sorting by price using any of the following methods:

* **`provider.sort` parameter** -- set `sort` to `"price"` in the `provider` object of your request body. See [Provider Sorting](/docs/guides/routing/provider-selection#provider-sorting) for details.
* **`:floor` virtual variant** -- append `:floor` to any model slug (e.g. `openai/gpt-4o:floor`) to sort by price. See [Floor Price Shortcut](/docs/guides/routing/provider-selection#floor-price-shortcut).
* **Default sort in account settings** -- set your default provider sort to price in your [account settings](https://openrouter.ai/settings/preferences) to apply price sorting across all requests.

Any of these will bypass Auto Exacto and return to the standard price-weighted provider ordering.