ORCFLO Index

Model Evaluation: o4-mini — May 10, 2026

The ORCFLO Indexis an independent benchmark that evaluates large language models the way business professionals actually use them — across real-world tasks spanning analysis, writing, extraction, summarization, and behavioral reliability. Each model is scored on three dimensions (quality, cost, and speed) by a panel of four independent judges. This report evaluates o4-mini in the context of 32 models from ORCFLO Index — May 10, 2026 cohort · 32 total models tested · Anthropic, Google, OpenAI, Mistral.

OpenAIOn ORCFLO

o4-mini

The Bottom Line

o4-mini lands at #17 of 32 with a quality score of 85.92, placing it firmly in the middle of the field well behind frontier leaders like GPT 5 (95.04), Gemini 3 Pro (93.17), and Claude Opus 4.6 (92.78). Its profile is uneven: strong on Summarization (#8) and Instruction Following (#13), but markedly weaker on Analysis (#23) and Writing (#20). At $0.0092 per case (#21 on cost), it is not a value pick — buyers pay mid-tier prices for mid-tier output, partially offset by respectable 7.2s response times (#12).

Quality

85.9

#17 of 323-Contender

-0.0 vs median · -9.1 from #1

Cost

1.4×median

$0.0092 per case

#21 of 323-Premium

107× cheapest in field

Speed

0.7×median

7.2s per case

#12 of 322-Quick

3.0× fastest in field

Key Findings

Summarization is the standout capability at 95.0 (#8 of 32, leader tier), placing o4-mini in the top quartile for condensation tasks despite a middling overall rank.
Analysis performance is a clear weak spot at 78.2 (#23 of 32), trailing nearly all quality peers and undermining the model's case for reasoning-heavy workloads.
Cost-to-quality ratio is unfavorable — at $0.0092 per case (#21), o4-mini is more expensive than Claude Haiku 4.5 ($0.0031) and GPT 4.1 ($0.0052), both of which score comparably or higher overall.
Speed is the redeeming operational trait at 7.2 seconds (#12 of 32), notably faster than same-tier peers o3 (13.5s) and Claude Sonnet 4.6 (13.7s).

Model Performance: Quality & Cost

The chart below plots quality against cost for all 32 models in the ORCFLO Index. Each dot represents the average quality score a model achieved across the full basket of real-world business tasks, alongside the cost in credits to complete the entire test suite. Models in the upper-left quadrant deliver the highest quality at the lowest cost.

Figure 1. Quality vs. cost across all 32 models. Upper-left quadrant = highest value. o4-mini highlighted. P50 median lines shown on both axes.

Model Performance: Quality & Time Elapsed

Quality alone doesn’t tell the full story — response time determines whether a model is viable for time-sensitive workflows. The chart below plots each model’s quality score against the total time required to complete the test suite. Models in the upper-left deliver the best quality with the least delay.

Figure 2. Quality vs. response time across all 32 models. Upper-left quadrant = best performance. o4-mini highlighted.

Category Scorecard

The ORCFLO Indexevaluates models using real-world business tasks — not academic puzzles or synthetic benchmarks. Each test case is designed to expose specific differences in how models handle the work professionals actually do. Scores are averaged across each category and ranked independently across all 32 models.

o4-mini Performance by Category

Category	Score	Rank	Tier
Abilities — Core language tasks: what the model can produce when given a well-formed prompt.
AnalysisReasoning, strategic judgment, disqualifying-factor detection	78.2	#23	Contender
ExtractionField accuracy, null handling, format compliance, zero fabrication	88.0	#10	Strong
SummarizationCompression quality, key-point retention, length compliance	95.0	#8	Leader
WritingTone, structure, persuasion, audience adaptation	82.5	#20	Contender
Behaviors — How the model acts under pressure: reliability, compliance, and restraint.
HallucinationFabrication detection, factual grounding, source fidelity	88.4	#21	Contender
Instruction FollowingConstraint adherence, format compliance, multi-part directives	91.2	#13	Strong
Refusal CalibrationAppropriate refusal vs. over-refusal on legitimate requests	77.5	#18	Contender
Stability — Repeatability and predictability across identical inputs.
Output ConsistencyRun-to-run reproducibility, format stability, score variance	86.5	#22	Contender

Strengths and Cautions

Strengths

Top-10 Summarization quality (95.0, rank #8) makes o4-mini competitive with frontier models specifically for digest, brief, and condensation tasks.
Instruction Following at 91.2 (rank #13, strong tier) supports reliable execution of structured prompts and multi-step directives.
Latency advantage among peers — at 7.2 seconds, o4-mini is nearly twice as fast as o3 (13.5s) and Claude Sonnet 4.6 (13.7s), the two models immediately above it in quality rank.

Cautions

Analysis rank of #23 (78.2) places o4-mini in contender tier for reasoning workloads, well below alternatives at similar or lower price points.
Hallucination score of 88.4 (#21) and Output Consistency of 86.5 (#22) indicate weaker reliability than the overall quality rank suggests — a concern for regulated or high-stakes outputs.
Pricing is not justified by output — at $0.0092 per case, o4-mini costs roughly 3x Claude Haiku 4.5 ($0.0031) while delivering a marginally lower overall quality score.

Head-to-Head: Frontier Models

o4-mini is OpenAI’s mid-tier contender in the ORCFLO Index. The table below compares it against the top-performing models from each major provider. Tier assignments use 25% quartiles across the full 32-model field.

Frontier Model Comparison

Model	Quality Avg	Quality Rank	Cost Rank	Speed Rank
Claude Sonnet 4.6	88.2	#14	#24	#24
o3	86.2	#15	#26	#22
GPT 5 Nano	85.9	#16	#10	#30
o4-mini	85.9	#17	#21	#12
Claude Sonnet 4	85.5	#18	#17	#14
GPT 4.1	85.0	#19	#15	#10
Claude Haiku 4.5	84.7	#20	#13	#11

When to Use o4-mini

Best pickHigh-volume summarization of meetings, documents, or reports where the #8 Summarization score (95.0) directly applies.

Best pickStructured extraction workflows requiring fast turnaround — Extraction rank #10 plus 7.2s latency suit pipeline integration.

ConsiderMid-complexity instruction-driven automation where the 91.2 Instruction Following score offsets weaker analytical depth.

AvoidAnalytical or strategic reasoning tasks — at rank #23 in Analysis, multiple cheaper alternatives outperform here.

AvoidLong-form writing or content generation, where the #20 Writing rank trails most peers and frontier models substantially.

The ORCFLO Index

This evaluation covers 40 cases across 8 categories. All tasks are text-only and English-only. Code generation, multi-turn conversation, multimodal tasks, and agentic workflows are not tested. Each contestant is scored by a panel of four independent judges — Gemini 2.5 Pro, Claude Opus 4.7, GPT 5.5, and Mistral Large — with final scores averaged across all four. Cost and speed measurements reflect API pricing and latency as of the test date (May 10, 2026) and will change as providers update their offerings.

How We Test

The ORCFLO Indexevaluates large language models across three independent dimensions — quality, cost, and speed — using real-world business tasks designed to expose the differences that matter for model selection. Each model is scored by a panel of four independent judges to reduce single-model bias.

Test Cases: 40 cases across 8 categories spanning Abilities (Analysis, Extraction, Summarization, Writing), Behaviors (Hallucination, Instruction Following, Refusal Calibration), and Stability (Output Consistency).
Judge Panel: Gemini 2.5 Pro, Claude Opus 4.7, GPT 5.5, and Mistral Large. Each judge scores independently. Final score is the average across all four.
Scoring: Three independent ranks: quality (higher is better), cost (lower is better), speed (faster is better). No composite score — composites hide the tradeoffs that drive model-selection decisions.
Tier Definitions: Leader
Quality ≥ 90.8
Ranks 1–8
Strong
≥ 85.9
Ranks 9–16
Contender
≥ 80.9
Ranks 17–24
Trailing
< 80.9
Ranks 25–32