OpenRouter 重建提供商路由算法，兼顾价格、速度与可靠性

Today we’re announcing the biggest improvement to our provider routing algorithm since the founding of OpenRouter.

Until now, our default router mostly rewarded providers for one thing: a low price. The new router rewards the inference that is actually best for you: cheap, but also fast, high-throughput, and reliable. Providers that deliver more value automatically win more traffic, so every provider has a reason to keep improving. That means the inference you get through OpenRouter keeps getting faster, more reliable, and more cost-effective, and you don’t have to change a line of code.

It’s already working. The new algorithm has unlocked significant improvements in TTFT (16.9%), throughput (16.3%) and capacity errors (a remarkable 67% reduction!), with only minor increases in cost (6.0%). That’s before providers have had any time to optimize for the new algorithm. See our full experimental results for more details.

How it works:

It measures what you actually get. For every model, it tracks each endpoint’s realized price, latency, throughput, and error rate, and picks the best blend of cost and performance for each request.
It reacts in real time. When an endpoint has trouble, traffic moves away from it until it recovers.
It rewards better inference. Providers that innovate are automatically rewarded with more traffic.

Routing is still cache-aware, so once we’ve sent you to a provider, your session stays put as long as the cache is warm.

Want price-only routing? You can still have it. See pinning and sorting providers in What stays the same.

Where we were: the limitations of Price-Only Routing

Most traffic (~77% of requests) on OpenRouter goes to models served by more than one provider. When a request doesn’t include an explicit sort or provider list, we need to pick a provider to send it to. Until now, we made that choice based almost entirely on input token price. The less expensive a provider was, the greater percentage of traffic it would receive, and any failed request was re-routed to the next cheapest option.

This price-only approach got users the lowest prices in the ecosystem, but it created some unintended consequences:

Cheap won even when it was bad. Fallbacks caught failures, but users waited through them, and speed improvements weren’t rewarded.
Static rankings. The algorithm was blind to real-time performance variations, so a provider had to degrade to the point of being de-ranked entirely before it was reflected in traffic distribution.
Vanishing margins. The only way for providers to win traffic was to cut price, putting pressure on them to reduce their investments in the very infrastructure that ensures reliability and quality.
Pricing games. The price we ranked by was almost all prompt input price, so a provider could keep traffic share with weird pricing cocktails like listing an artificially low input price and then making back the difference with overpriced cached or completion tokens.

We wanted to ensure a marketplace where competing providers won traffic by serving better inference, not just cheaper inference.

The New Algorithm: optimizing for what a good answer “costs”

The new routing algorithm constantly monitors the performance of each provider to calculate a real-time Routing Cost that captures what a user pays for a good answer. The Routing Cost encompasses the price in actual dollars and cents, the cost of time spent waiting for a response, and the cost of retries when a provider fails or provides an unusable response.

To put it simply: what does one try cost, and how often does it work?

Diagram: cost of one try, $0.40 price plus 0.60 waiting, is 1.00. Divided by an 80% chance it works, that is a Routing Cost of 1.25 per good answer. Lower wins more traffic. (light mode)

Diagram: cost of one try, $0.40 price plus 0.60 waiting, is 1.00. Divided by an 80% chance it works, that is a Routing Cost of 1.25 per good answer. Lower wins more traffic. (dark mode)

What one try costs: price plus waiting

For each request, the router estimates how many input and output tokens it will use from the model’s recent traffic. That one estimate sets both parts of a try: the price you would pay and the time you would wait.

Effective price

Effective price is the price that the provider would actually charge you, calculated from recent measured traffic. It reflects the provider’s real cache hit rate and the model’s average mix of input and output tokens, so it can differ a lot from the list price. This ensures that a provider can’t win traffic by hiding expensive cached or output tokens behind a low input price.

Chart of average input and output tokens per request by prompt size, on log scales. As prompts grow from about 370 to 670K input tokens, output only grows from about 190 to 1,050 tokens, so input goes from 2 times output to 8 times at 5.7K input tokens, 90 times at 93K, and 630 times at 670K. (light mode)

Effective price example: 40K uncached input tokens at $2.00/M is $0.080, 60K cached input tokens at $0.50/M is $0.030, and about 4K output tokens at $8.00/M is $0.032, for an effective price of $0.142 for one request. (light mode)

The router prices your request at each provider’s effective price, and prioritizes the cheapest options.

Throughput

Throughput is what trades off against price: a faster provider can charge more and still win your request. It’s measured as output tokens per second end-to-end, so it includes the time to first token plus the time it takes the provider to stream the entire output response. The router adds that wait to the Routing Cost using a conversion rate set per model, based on what its providers typically charge and how fast they are.

Throughput example: a 5 second time to first token plus 500 tokens streamed in 5 seconds is 50 tokens per second end to end, with the time to first token included. (light mode)

Throughput example: a 5 second time to first token plus 500 tokens streamed in 5 seconds is 50 tokens per second end to end, with the time to first token included. (dark mode)

Time to first token

Time to first token (TTFT) is how long you would wait before the response started streaming. It adds to the Routing Cost on its own, even though it already counts toward throughput, because interactive use depends on it most. In an agent session, the model may make many tool calls in a row, and each one waits for a first token.

An agent session: reasoning, then five tool calls, then more reasoning and the response. Each step waits for a first token, so the tool calls are mostly waiting on time to first token. (light mode)

An agent session: reasoning, then five tool calls, then more reasoning and the response. Each step waits for a first token, so the tool calls are mostly waiting on time to first token. (dark mode)

Rate limits

A rate limit (HTTP 429) means the provider is out of capacity and turned your request away. You would have to wait for the rejection, then your full request would go to another provider. The router adds that wait plus a penalty to the provider’s Routing Cost, weighted by how often it rate limits requests, so the effect is usually small. Repeated rate limits matter more through the traffic caps and backoff described below.

A 429 makes you wait for the rejection, then your full request goes elsewhere. Example: a 2.50 wait plus penalty times a 0.4% 429 rate adds 0.01 to the Routing Cost. (light mode)

A 429 makes you wait for the rejection, then your full request goes elsewhere. Example: a 2.50 wait plus penalty times a 0.4% 429 rate adds 0.01 to the Routing Cost. (dark mode)

How often it works: reliability and quality

A failed try still costs its price and its wait, so the Routing Cost calculates the expected cost of having to retry until one works. If 1 in 5 tries fails, you pay for 1.25 tries per good answer.

Reliability

A reliability error is a request that fails because of the provider. You would lose the attempt and have to retry, so each error carries a heavy penalty in the Routing Cost.

Reliability example: a 1% error rate times an 8x penalty per error raises the Routing Cost by about 8%. (light mode)

Reliability example: a 1% error rate times an 8x penalty per error raises the Routing Cost by about 8%. (dark mode)

If a provider’s error rate climbs far above the other providers for the same model, the router stops sending it requests until it recovers.

Quality

Quality measures whether a provider’s responses are usable. Today the main check is whether tool calls come back well formed, and we’re adding more. Each bad response raises the provider’s Routing Cost, because a cheap wrong answer is expensive.

Quality example: 95% of tool calls well formed raises the Routing Cost by about 5%. (light mode)

Quality example: 95% of tool calls well formed raises the Routing Cost by about 5%. (dark mode)

A provider flagged for broken tool calls, or for benchmark accuracy well below what the model should score, drops far behind and only gets a small amount of test traffic until the flag clears.

Sending every request to the single best provider would overload it, so traffic is shared. Faster providers can carry a higher Routing Cost and stay competitive. Everything inside the competitive edge shares most of the traffic, and being a little behind the best costs a little. Past the edge, traffic share drops off fast.

Three guardrails sit on top:

Diagram of requests flowing to three providers. A is at its cap, so the extra spills over to other providers. B is struggling, so it gets less. C is furthest behind but still gets a trickle. (light mode)

Floor: every provider keeps a trickle of traffic, so we can notice when struggling endpoints improve. New endpoints start from the model’s typical numbers, so they don’t need a traffic history.
Back off: a provider that starts failing or rate limiting gets less until it recovers.
Cap: the router learns how much load each provider can take before it returns 429s, and spills the rest to others. Add capacity and the cap rises on its own.

What stays the same

Requests that set provider.order bypass the new routing logic, with providers ordered as specified (default is price-based as before). Setting provider.only or provider.ignore will use the new algorithm to select from the remaining providers if there are more than one.
An explicit provider.sort (price, throughput, or latency) uses that sort instead of the new algorithm.
Only the first request in a conversation will have a provider algorithmically selected. An ongoing conversation is pinned to its original provider so that the prompt cache stays warm. If that provider becomes unavailable, the conversation will fall back to another provider automatically.
If a try fails, the request is retried with the next highest ranked provider.

How we measured it

In order to measure the impact of the new routing algorithm, we had to go beyond a standard A/B test, because changes in routing behavior can’t easily be isolated from the control group. For example, if the new algorithm did a better job of routing traffic away from a struggling provider, the old algorithm’s requests to that provider would find more capacity available and get faster, skewing the results. Further compounding the difficulty is the enormous variability in request shape and volume patterns. A single whale user that spins up an automated flurry of high effort requests to a single provider can artificially inflate the metrics for that provider regardless of which algorithm we used to route their requests.

So we ran a switchback experiment: We split each day into 3-hour blocks and flipped each matched pair of blocks between the new and old router, with a one-hour burn-in period of discarded data each changeover, and measured the per-model impact across a representative sample of open source models over one full week.

Results

Over the week of the experiment, we split over 24 million requests with 103 billion input tokens evenly between the old and new routing algorithms, measuring the impact on key performance metrics.

Measure	Legacy	New Algorithm	New vs legacy	95% interval
TTFT p50	1,007 ms	837 ms	16.9% lower	13.9% to 19.8% lower
TTFT p95	4,601 ms	3,733 ms	18.9% lower	15.7% to 21.9% lower
Throughput p50	62.4 tok/s	72.6 tok/s	16.3% higher	13.8% to 18.9% higher
Cost per K requests	$1.25	$1.32	6.0% higher	5.1% to 7.0% higher
Cost per M tokens	$0.301	$0.324	7.3% higher	4.9% to 9.6% higher
Caller TTFT p50	1,130 ms	941 ms	16.7% lower	13.8% to 19.7% lower
Caller TTFT p95	6,032 ms	4,504 ms	25.3% lower	21.6% to 29.0% lower
Whole request p50 (time to last token)	4.00 s	3.24 s	19.0% lower	14.2% to 23.6% lower
Whole request p95	41.5 s	34.9 s	16.0% lower	12.2% to 19.7% lower
Hit at least one 429	7.1%	3.8%	−3.4 pt 67% fewer	−4.2 to −2.5 pt 62% to 72% fewer
Needed a retry	10.4%	6.9%	−3.5 pt 47% fewer	−4.4 to −2.6 pt 41% to 52% fewer
Time spent in failed attempts, per req	349 ms	242 ms	−106 ms 55% less	−181 to −35 ms 48% to 60% less

Get started

If you use default routing, you’re done. The provider that served each request is in the response and on your activity page.

To keep the old price-only routing, set provider.sort to "price", append :floor to the model slug, or set your default sort to price in account settings. To pick providers yourself, use provider.order or provider.only. See the provider routing guide.

Frequently asked questions

Do I need to change my code to use the new routing?

No. Requests with no explicit provider sort and no provider order, only, or ignore list use the new router automatically on the models where it's enabled. Your API key, request body, and model IDs stay the same.

How do I keep price-only routing?

Set provider.sort to price in the request body, append :floor to the model slug, or set your default provider sort to price in account settings. Any explicit sort bypasses the new router.

Does the new routing override my provider order or only list?

provider.order bypasses the new router and routes in the order you specify. provider.only and provider.ignore still filter the candidate set, but the new algorithm picks among the providers that remain.

Will the new routing cost more than price-only routing?

It can, since it will sometimes pick a faster or more reliable provider over the cheapest one. In our live experiment, cost per request increased by 6.0%, while realizing a 16.9% improvement in TTFT, a 16.3% improvement in throughput, and a 67% decrease in 429 error rate.

How does a provider get more traffic under the new routing?

Improve the signals the router scores: effective price, time to first token, throughput, error and rate-limit rates, and output quality.