Results · pricing-intelligence-research-001 · the workload

The Flagship Fight

Real OpenRouter executions. 3 runs per model, medians reported, every failure preserved.

executed model test 10 contestants 3 runs each

Cheapest accepted result

xAI: Grok 4.5

x-ai/grok-4.5 · xAI

$6.26

landed cost per accepted result

Leaderboard

landed cost · runtime · acceptance · repairs · evidence

1 xAI: Grok 4.5 xAI · x-ai/grok-4.5 $6.26 12.8s 100% 0 3/3

Clean sweep — accepted on every first attempt.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-x-ai-grok-4-5-001	accepted	100/100	$0.0092 provider-reported	—	evidence
run-real-flagship-fight-x-ai-grok-4-5-002	accepted	100/100	$0.0096 provider-reported	—	evidence
run-real-flagship-fight-x-ai-grok-4-5-003	accepted	100/100	$0.0073 provider-reported	—	evidence

2 MoonshotAI: Kimi K2.5 Moonshot AI · moonshotai/kimi-k2.5 $6.26 88.8s 67% 0 2/3

Accepted 2 of 3 runs.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-moonshot-kimi-k2-5-001	accepted	100/100	$0.0136 provider-reported	—	evidence
run-real-flagship-fight-moonshot-kimi-k2-5-002	failed	89/100	$0.0114 provider-reported	—	evidence
run-real-flagship-fight-moonshot-kimi-k2-5-003	accepted	100/100	$0.0114 provider-reported	—	evidence

3 Google: Gemini 3.1 Pro (Preview) Google · google/gemini-3.1-pro-preview $6.38 45.4s 100% ×0.667 3/3

Accepted every run, but needed schema repair to get there.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-google-gemini-3-1-pro-preview-001	accepted	100/100	$0.0932 provider-reported	—	evidence
run-real-flagship-fight-google-gemini-3-1-pro-preview-002	accepted	100/100	$0.0455 provider-reported	—	evidence
run-real-flagship-fight-google-gemini-3-1-pro-preview-003	accepted	100/100	$0.0942 provider-reported	—	evidence

— OpenAI: GPT-5.1 OpenAI · openai/gpt-5.1 DNF 15.6s 0% 0 0/3

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-openai-gpt-5-1-001	failed	58/100	$0.0163 provider-reported	—	evidence
run-real-flagship-fight-openai-gpt-5-1-002	failed	53/100	$0.0231 provider-reported	—	evidence
run-real-flagship-fight-openai-gpt-5-1-003	failed	53/100	$0.0147 provider-reported	—	evidence

— Anthropic: Claude Opus 4.5 Anthropic · anthropic/claude-opus-4.5 DNF 8.7s 0% 0 0/3

Failed every attempt — incorrect winning option. Evidence preserved below.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-anthropic-claude-opus-4-5-001	failed	89/100	$0.0258 provider-reported	—	evidence
run-real-flagship-fight-anthropic-claude-opus-4-5-002	failed	89/100	$0.0299 provider-reported	—	evidence
run-real-flagship-fight-anthropic-claude-opus-4-5-003	failed	89/100	$0.0238 provider-reported	—	evidence

— DeepSeek: DeepSeek V3.2 DeepSeek · deepseek/deepseek-v3.2 DNF 21.3s 0% ×0.333 0/3

Failed every attempt — incorrect winning option. Evidence preserved below.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-deepseek-deepseek-v3-2-001	failed	89/100	$0.0019 provider-reported	—	evidence
run-real-flagship-fight-deepseek-deepseek-v3-2-002	failed	58/100	$0.0004 provider-reported	—	evidence
run-real-flagship-fight-deepseek-deepseek-v3-2-003	failed	58/100	$0.0004 provider-reported	—	evidence

— Alibaba: Qwen3 Max Alibaba Qwen · qwen/qwen3-max DNF 37.0s 0% 0 0/3

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-qwen-qwen3-max-001	failed	53/100	$0.0045 provider-reported	—	evidence
run-real-flagship-fight-qwen-qwen3-max-002	failed	53/100	$0.0120 provider-reported	—	evidence
run-real-flagship-fight-qwen-qwen3-max-003	failed	53/100	$0.0109 provider-reported	—	evidence

— Mistral: Mistral Large 2512 Mistral AI · mistralai/mistral-large-2512 DNF 9.2s 0% 0 0/3

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-mistralai-mistral-large-2512-001	failed	58/100	$0.0019 provider-reported	—	evidence
run-real-flagship-fight-mistralai-mistral-large-2512-002	failed	58/100	$0.0020 provider-reported	—	evidence
run-real-flagship-fight-mistralai-mistral-large-2512-003	failed	58/100	$0.0014 provider-reported	—	evidence

— Meta: Llama 4 Maverick Meta · meta-llama/llama-4-maverick DNF 25.1s 0% 0 0/3

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-meta-llama-4-maverick-001	failed	58/100	$0.0011 provider-reported	—	evidence
run-real-flagship-fight-meta-llama-4-maverick-002	failed	58/100	$0.0008 provider-reported	—	evidence
run-real-flagship-fight-meta-llama-4-maverick-003	failed	58/100	$0.0009 provider-reported	—	evidence

— Cohere: Command A Cohere · cohere/command-a DNF 12.1s 0% ×0.667 0/3

Failed every attempt — malformed required output. Evidence preserved below.

Run	Status	Score	API cost	Repair	Evidence
run-real-flagship-fight-cohere-command-a-001	invalid	0/100	$0.0000 provider-reported	—	evidence
run-real-flagship-fight-cohere-command-a-002	failed	58/100	$0.0126 provider-reported	—	evidence
run-real-flagship-fight-cohere-command-a-003	failed	58/100	$0.0089 provider-reported	—	evidence

Cheapest accepted result

xAI: Grok 4.5

6.2592 USD/accepted result

Most reliable

Google: Gemini 3.1 Pro (Preview)

1 eventual acceptance rate

Fastest accepted

xAI: Grok 4.5

12752 median runtime ms

Best first attempt

Google: Gemini 3.1 Pro (Preview)

1 first-attempt acceptance rate

Highest landed cost

Google: Gemini 3.1 Pro (Preview)

6.3805 USD/accepted result

Largest overpayment gap

Google: Gemini 3.1 Pro (Preview)

0.1213 USD above cheapest accepted

Best evidence discipline

Google: Gemini 3.1 Pro (Preview)

1 source-grounding success rate

Model profiles

xAI: Grok 4.5

xAI · x-ai/grok-4.5

accepted

Cost / accepted: $6.26
Overpayment gap: +$0.00
Acceptance: 100% (3/3)
Median runtime: 12.8s
Median tokens in/out: 1493 / 1126
Evidence discipline: 100%

Clean sweep — accepted on every first attempt.

Tested 2026-07-30 · real

MoonshotAI: Kimi K2.5

Moonshot AI · moonshotai/kimi-k2.5

accepted

Cost / accepted: $6.26
Overpayment gap: +$0.00
Acceptance: 67% (2/3)
Median runtime: 88.8s
Median tokens in/out: 1219 / 4199
Evidence discipline: 100%

Accepted 2 of 3 runs.

Tested 2026-07-30 · real

Google: Gemini 3.1 Pro (Preview)

Google · google/gemini-3.1-pro-preview

accepted

Cost / accepted: $6.38
Overpayment gap: +$0.12
Acceptance: 100% (3/3)
Median runtime: 45.4s
Median tokens in/out: 2998 / 7268
Evidence discipline: 100%

Accepted every run, but needed schema repair to get there.

Tested 2026-07-30 · real

OpenAI: GPT-5.1

OpenAI · openai/gpt-5.1

failed

Cost / accepted: —
Overpayment gap: —
Acceptance: 0% (0/3)
Median runtime: 15.6s
Median tokens in/out: 1223 / 1473
Evidence discipline: 100%

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Tested 2026-07-30 · real

Anthropic: Claude Opus 4.5

Anthropic · anthropic/claude-opus-4.5

failed

Cost / accepted: —
Overpayment gap: —
Acceptance: 0% (0/3)
Median runtime: 8.7s
Median tokens in/out: 1419 / 747
Evidence discipline: 100%

Failed every attempt — incorrect winning option. Evidence preserved below.

Tested 2026-07-30 · real

DeepSeek: DeepSeek V3.2

DeepSeek · deepseek/deepseek-v3.2

failed

Cost / accepted: —
Overpayment gap: —
Acceptance: 0% (0/3)
Median runtime: 21.3s
Median tokens in/out: 1259 / 717
Evidence discipline: 100%

Failed every attempt — incorrect winning option. Evidence preserved below.

Tested 2026-07-30 · real

Alibaba: Qwen3 Max

Alibaba Qwen · qwen/qwen3-max

failed

Cost / accepted: —
Overpayment gap: —
Acceptance: 0% (0/3)
Median runtime: 37.0s
Median tokens in/out: 1316 / 2731
Evidence discipline: 100%

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Tested 2026-07-30 · real

Mistral: Mistral Large 2512

Mistral AI · mistralai/mistral-large-2512

failed

Cost / accepted: —
Overpayment gap: —
Acceptance: 0% (0/3)
Median runtime: 9.2s
Median tokens in/out: 1345 / 860
Evidence discipline: 100%

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Tested 2026-07-30 · real

Meta: Llama 4 Maverick

Meta · meta-llama/llama-4-maverick

failed

Cost / accepted: —
Overpayment gap: —
Acceptance: 0% (0/3)
Median runtime: 25.1s
Median tokens in/out: 1222 / 696
Evidence discipline: 100%

Failed every attempt — unreconciled arithmetic. Evidence preserved below.

Tested 2026-07-30 · real

Cohere: Command A

Cohere · cohere/command-a

failed

Cost / accepted: —
Overpayment gap: —
Acceptance: 0% (0/3)
Median runtime: 12.1s
Median tokens in/out: 2396 / 553
Evidence discipline: 67%

Failed every attempt — malformed required output. Evidence preserved below.

Tested 2026-07-30 · real

Failed-run evidence

Every failure is preserved with its raw request, raw response, validation output, and SHA-256 evidence hash.

OpenAI: GPT-5.1

unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, incorrect_winning_option, unreconciled_arithmetic, unreconciled_arithmetic

openai/gpt-5.1

unreconciled_arithmetic — repeated across 3 runs. Inspect manifests.

Anthropic: Claude Opus 4.5

incorrect_winning_option, unreconciled_arithmetic

anthropic/claude-opus-4.5

incorrect_winning_option — repeated across 3 runs. Inspect manifests.

DeepSeek: DeepSeek V3.2

incorrect_winning_option, unreconciled_arithmetic

deepseek/deepseek-v3.2

incorrect_winning_option — repeated across 3 runs. Inspect manifests.

Alibaba: Qwen3 Max

unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, incorrect_winning_option, unreconciled_arithmetic, unreconciled_arithmetic

qwen/qwen3-max

unreconciled_arithmetic — repeated across 3 runs. Inspect manifests.

Mistral: Mistral Large 2512

unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, incorrect_winning_option, unreconciled_arithmetic, unreconciled_arithmetic

mistralai/mistral-large-2512

unreconciled_arithmetic — repeated across 3 runs. Inspect manifests.

Meta: Llama 4 Maverick

unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, unreconciled_arithmetic, incorrect_winning_option, unreconciled_arithmetic, unreconciled_arithmetic

meta-llama/llama-4-maverick

unreconciled_arithmetic — repeated across 3 runs. Inspect manifests.

Cohere: Command A

malformed_required_output

cohere/command-a

malformed_required_output — repeated across 3 runs. Inspect manifests.

Methodology & limitations

repeat Policy: 3 runs per contestant. Medians reported. No best-run cherry picking. All failures preserved.
repair Policy: Maximum 1 schema-repair attempt per run.
validation: Deterministic pure-Python validation. No LLM judges arithmetic.
cost Basis: Provider-reported cost where available; otherwise calculated from catalogue pricing. Every cost is labelled with its basis.
fixture Vs Real: Fixture runs use canned responses and zero spend. Fixture data is labelled and must not be cited as benchmark evidence.

Limitations

Frozen pricing sources captured 2026-07-30; live prices may differ.
Single prompt, no live browsing — models answer from frozen evidence only.
Landed cost uses a fixed human-review rate ($75/hr, 5 min/result).

Excluded at discovery (6)

openai/gpt-5.1-mini — excluded pattern 'mini'
openai/gpt-5.1-codex — excluded pattern 'codex'
openai/text-embedding-3-large — excluded pattern 'embed'
google/gemini-3-flash — excluded pattern 'flash'
microsoft/phi-4 — context window 16,000 below minimum 32,000
Microsoft — no eligible flagship found in catalogue