ORCFLO Index

Model Evaluation: Mistral Large 3 (2512) — May 10, 2026

The ORCFLO Indexis an independent benchmark that evaluates large language models the way business professionals actually use them — across real-world tasks spanning analysis, writing, extraction, summarization, and behavioral reliability. Each model is scored on three dimensions (quality, cost, and speed) by a panel of four independent judges. This report evaluates Mistral Large 3 (2512) in the context of 32 models from ORCFLO Index — May 10, 2026 cohort · 32 total models tested · Anthropic, Google, OpenAI, Mistral.

MistralOn ORCFLO

Mistral Large 3 (2512)

The Bottom Line

Mistral Large 3 (2512) lands at #22 of 32 on overall quality (83.22), placing it in the mid-tier contender band — well behind frontier leaders like GPT 5 (95.04) and Gemini 3 Pro (93.17), and trailing peer offerings from Anthropic and OpenAI in the high-80s. Its strongest showing is summarization (#10), but extraction, hallucination control, and output consistency all rank #25. Cost efficiency at $0.0015/case (#11) is the main reason to consider it, though response time of 13.4s (#21) undercuts that advantage.

Quality

83.2

#22 of 323-Contender

-2.7 vs median · -11.8 from #1

Cost

0.2×median

$0.0015 per case

#11 of 322-Standard

17× cheapest in field

Speed

1.3×median

13.4s per case

#21 of 323-Moderate

5.6× fastest in field

Key Findings

Summarization is the standout capability at 93.7 (rank #10), the only category where Large 3 cracks the top third of the 32-model field.
Reliability metrics are a concern: hallucination (81.5, #25), extraction (81.4, #25), and output consistency (84.0, #25) all sit in the trailing tier.
Cost efficiency is competitive but not exceptional at $0.0015/case (#11) — cheaper peers Gemini 2.0 Flash and 2.5 Flash-Lite deliver similar or better quality at $0.0003.
Speed is a weak point at 13.4s average response (#21), notably slower than peer-band models GPT 4.1 (5.2s) and Claude Haiku 4.5 (5.2s).

Model Performance: Quality & Cost

The chart below plots quality against cost for all 32 models in the ORCFLO Index. Each dot represents the average quality score a model achieved across the full basket of real-world business tasks, alongside the cost in credits to complete the entire test suite. Models in the upper-left quadrant deliver the highest quality at the lowest cost.

Figure 1. Quality vs. cost across all 32 models. Upper-left quadrant = highest value. Mistral Large 3 (2512) highlighted. P50 median lines shown on both axes.

Model Performance: Quality & Time Elapsed

Quality alone doesn’t tell the full story — response time determines whether a model is viable for time-sensitive workflows. The chart below plots each model’s quality score against the total time required to complete the test suite. Models in the upper-left deliver the best quality with the least delay.

Figure 2. Quality vs. response time across all 32 models. Upper-left quadrant = best performance. Mistral Large 3 (2512) highlighted.

Category Scorecard

The ORCFLO Indexevaluates models using real-world business tasks — not academic puzzles or synthetic benchmarks. Each test case is designed to expose specific differences in how models handle the work professionals actually do. Scores are averaged across each category and ranked independently across all 32 models.

Mistral Large 3 (2512) Performance by Category

Category	Score	Rank	Tier
Abilities — Core language tasks: what the model can produce when given a well-formed prompt.
AnalysisReasoning, strategic judgment, disqualifying-factor detection	82.2	#19	Contender
ExtractionField accuracy, null handling, format compliance, zero fabrication	81.4	#25	Trailing
SummarizationCompression quality, key-point retention, length compliance	93.7	#10	Strong
WritingTone, structure, persuasion, audience adaptation	82.8	#19	Contender
Behaviors — How the model acts under pressure: reliability, compliance, and restraint.
HallucinationFabrication detection, factual grounding, source fidelity	81.5	#25	Trailing
Instruction FollowingConstraint adherence, format compliance, multi-part directives	87.2	#22	Contender
Refusal CalibrationAppropriate refusal vs. over-refusal on legitimate requests	72.9	#21	Contender
Stability — Repeatability and predictability across identical inputs.
Output ConsistencyRun-to-run reproducibility, format stability, score variance	84.0	#25	Trailing

Strengths and Cautions

Strengths

Summarization at rank #10 (93.7) makes it viable for document condensation, meeting recaps, and report compression workloads.
Mid-pack cost positioning at $0.0015/case (#11 of 32) keeps it competitive on per-call economics versus pricier peers like Claude Sonnet 4.5 ($0.0088).
Adequate instruction following at 87.2 (#22) supports routine structured tasks where the workflow can tolerate occasional drift.

Cautions

Extraction and hallucination both rank #25 (81.4 and 81.5) — a poor combination for any workflow involving factual retrieval, data parsing, or knowledge-base grounding.
Output consistency at #25 (84.0) signals variability that complicates production deployments where reproducibility matters.
Speed of 13.4 seconds (#21) is roughly 2.5x slower than similarly-priced peers Gemini 2.0 Flash (4.0s) and GPT 4.1 Mini (5.1s), eroding its cost advantage in interactive use cases.

Head-to-Head: Frontier Models

Mistral Large 3 (2512) is Mistral’s mid-tier contender in the ORCFLO Index. The table below compares it against the top-performing models from each major provider. Tier assignments use 25% quartiles across the full 32-model field.

Frontier Model Comparison

Model	Quality Avg	Quality Rank	Cost Rank	Speed Rank
GPT 4.1	85.0	#19	#15	#10
Claude Haiku 4.5	84.7	#20	#13	#11
Claude Sonnet 4.5	84.6	#21	#20	#17
Mistral Large 3 (2512)	83.2	#22	#11	#21
Gemini 2.0 Flash	83.1	#23	#5	#4
Gemini 2.5 Flash-Lite	80.9	#24	#4	#5
GPT 4.1 Mini	80.4	#25	#9	#9

When to Use Mistral Large 3 (2512)

Best pickBatch summarization of long documents, transcripts, or reports where the #10 summarization score delivers genuine value.

Best pickCost-sensitive content compression workloads that run asynchronously and can absorb the 13-second latency.

ConsiderGeneral-purpose writing or analysis tasks where the mid-tier scores (~82) are sufficient and Mistral vendor diversification is a procurement priority.

AvoidData extraction, structured parsing, or any factual-grounding workflow — extraction and hallucination both rank #25 of 32.

AvoidReal-time or interactive applications where 13.4-second response times will degrade user experience versus sub-5-second peers.

The ORCFLO Index

This evaluation covers 40 cases across 8 categories. All tasks are text-only and English-only. Code generation, multi-turn conversation, multimodal tasks, and agentic workflows are not tested. Each contestant is scored by a panel of four independent judges — Gemini 2.5 Pro, Claude Opus 4.7, GPT 5.5, and Mistral Large — with final scores averaged across all four. Cost and speed measurements reflect API pricing and latency as of the test date (May 10, 2026) and will change as providers update their offerings.

How We Test

The ORCFLO Indexevaluates large language models across three independent dimensions — quality, cost, and speed — using real-world business tasks designed to expose the differences that matter for model selection. Each model is scored by a panel of four independent judges to reduce single-model bias.

Test Cases: 40 cases across 8 categories spanning Abilities (Analysis, Extraction, Summarization, Writing), Behaviors (Hallucination, Instruction Following, Refusal Calibration), and Stability (Output Consistency).
Judge Panel: Gemini 2.5 Pro, Claude Opus 4.7, GPT 5.5, and Mistral Large. Each judge scores independently. Final score is the average across all four.
Scoring: Three independent ranks: quality (higher is better), cost (lower is better), speed (faster is better). No composite score — composites hide the tradeoffs that drive model-selection decisions.
Tier Definitions: Leader
Quality ≥ 90.8
Ranks 1–8
Strong
≥ 85.9
Ranks 9–16
Contender
≥ 80.9
Ranks 17–24
Trailing
< 80.9
Ranks 25–32