ORCFLO Index

Model Evaluation: GPT 4o Mini — May 10, 2026

The ORCFLO Indexis an independent benchmark that evaluates large language models the way business professionals actually use them — across real-world tasks spanning analysis, writing, extraction, summarization, and behavioral reliability. Each model is scored on three dimensions (quality, cost, and speed) by a panel of four independent judges. This report evaluates GPT 4o Mini in the context of 32 models from ORCFLO Index — May 10, 2026 cohort · 32 total models tested · Anthropic, Google, OpenAI, Mistral.

OpenAIOn ORCFLO

GPT 4o Mini

The Bottom Line

GPT 4o Mini ranks #30 of 32 on overall quality (72.82), placing it firmly in the trailing tier well behind frontier models like GPT 5 (95.04) and Gemini 3 Pro (93.17). Its appeal is economic, not qualitative: at $0.0003 per case it ranks #6 on cost, with #8 speed (4.8s). The model trails in seven of eight evaluated categories. Suitable only for high-volume, low-complexity workloads where price dominates the decision.

Quality

72.8

#30 of 324-Trailing

-13.1 vs median · -22.2 from #1

Cost

0.0×median

$0.0003 per case

#6 of 321-Budget

3.8× cheapest in field

Speed

0.5×median

4.8s per case

#8 of 321-Fast

2.0× fastest in field

Key Findings

Bottom-tier quality across abilities and behaviors: ranks #31 in Analysis (53.5), Extraction (77.1), Summarization (69.2), and Hallucination (74.0), with Refusal Calibration at #29 (60.1).
Cost is the primary justification: at $0.0003 per case (#6 of 32), it sits roughly 18x cheaper than o3-mini ($0.0154) and 50x cheaper than typical frontier-tier pricing.
Output Consistency is the lone bright spot: 92.8 score (#11 of 32, strong tier) — predictable formatting even when content quality lags.
Analysis score of 53.5 is a structural weakness: ranking #31 of 32, it should not be deployed where reasoning depth matters.

Model Performance: Quality & Cost

The chart below plots quality against cost for all 32 models in the ORCFLO Index. Each dot represents the average quality score a model achieved across the full basket of real-world business tasks, alongside the cost in credits to complete the entire test suite. Models in the upper-left quadrant deliver the highest quality at the lowest cost.

Figure 1. Quality vs. cost across all 32 models. Upper-left quadrant = highest value. GPT 4o Mini highlighted. P50 median lines shown on both axes.

Model Performance: Quality & Time Elapsed

Quality alone doesn’t tell the full story — response time determines whether a model is viable for time-sensitive workflows. The chart below plots each model’s quality score against the total time required to complete the test suite. Models in the upper-left deliver the best quality with the least delay.

Figure 2. Quality vs. response time across all 32 models. Upper-left quadrant = best performance. GPT 4o Mini highlighted.

Category Scorecard

The ORCFLO Indexevaluates models using real-world business tasks — not academic puzzles or synthetic benchmarks. Each test case is designed to expose specific differences in how models handle the work professionals actually do. Scores are averaged across each category and ranked independently across all 32 models.

GPT 4o Mini Performance by Category

Category	Score	Rank	Tier
Abilities — Core language tasks: what the model can produce when given a well-formed prompt.
AnalysisReasoning, strategic judgment, disqualifying-factor detection	53.5	#31	Trailing
ExtractionField accuracy, null handling, format compliance, zero fabrication	77.1	#31	Trailing
SummarizationCompression quality, key-point retention, length compliance	69.2	#31	Trailing
WritingTone, structure, persuasion, audience adaptation	70.8	#28	Trailing
Behaviors — How the model acts under pressure: reliability, compliance, and restraint.
HallucinationFabrication detection, factual grounding, source fidelity	74.0	#31	Trailing
Instruction FollowingConstraint adherence, format compliance, multi-part directives	84.9	#26	Trailing
Refusal CalibrationAppropriate refusal vs. over-refusal on legitimate requests	60.1	#29	Trailing
Stability — Repeatability and predictability across identical inputs.
Output ConsistencyRun-to-run reproducibility, format stability, score variance	92.8	#11	Strong

Strengths and Cautions

Strengths

Output Consistency #11 (92.8): the only category where the model reaches the strong tier, making it usable for templated, repeatable outputs.
Cost efficiency #6 ($0.0003/case): among the cheapest commercial options, undercutting GPT 4o ($0.0054) by roughly 18x at the same vendor.
Speed #8 (4.8s): top-quartile latency makes it viable for interactive, high-throughput pipelines where response time matters.

Cautions

Analysis rank #31 (53.5): nearly last in the cohort — avoid for tasks requiring multi-step reasoning, synthesis, or judgment.
Hallucination rank #31 (74.0): factual reliability is among the weakest in the field; not suitable for unsupervised customer-facing or compliance-sensitive output.
Refusal Calibration rank #29 (60.1): the model handles edge-case requests poorly, creating friction in production deployments.

Head-to-Head: Frontier Models

GPT 4o Mini is OpenAI’s budget value pick in the ORCFLO Index. The table below compares it against the top-performing models from each major provider. Tier assignments use 25% quartiles across the full 32-model field.

Frontier Model Comparison

Model	Quality Avg	Quality Rank	Cost Rank	Speed Rank
Mistral Small 3 (24B)	79.9	#26	#1	#7
Gemini 2.0 Flash-Lite	78.0	#27	#3	#2
o3-mini	77.4	#28	#27	#20
GPT 4o	75.4	#29	#16	#6
GPT 4o Mini	72.8	#30	#6	#8
GPT 4.1 Nano	71.3	#31	#2	#3
Codestral (2508)	71.1	#32	#8	#1

When to Use GPT 4o Mini

Best pickHigh-volume, templated text generation where unit cost (<$0.001/case) is the binding constraint.

Best pickLow-stakes formatting, classification, or tagging pipelines that benefit from #11 Output Consistency.

ConsiderInteractive chat prototypes where 4.8s latency and low cost matter more than answer depth — but A/B test against GPT 4.1 Nano ($0.0002) and Gemini 2.0 Flash-Lite ($0.0002).

AvoidAnalytical workloads, research synthesis, or strategic planning — Analysis rank #31 (53.5) is disqualifying.

AvoidCompliance, legal, medical, or factual-accuracy critical tasks — Hallucination rank #31 (74.0) is too high a risk.

The ORCFLO Index

This evaluation covers 40 cases across 8 categories. All tasks are text-only and English-only. Code generation, multi-turn conversation, multimodal tasks, and agentic workflows are not tested. Each contestant is scored by a panel of four independent judges — Gemini 2.5 Pro, Claude Opus 4.7, GPT 5.5, and Mistral Large — with final scores averaged across all four. Cost and speed measurements reflect API pricing and latency as of the test date (May 10, 2026) and will change as providers update their offerings.

How We Test

The ORCFLO Indexevaluates large language models across three independent dimensions — quality, cost, and speed — using real-world business tasks designed to expose the differences that matter for model selection. Each model is scored by a panel of four independent judges to reduce single-model bias.

Test Cases: 40 cases across 8 categories spanning Abilities (Analysis, Extraction, Summarization, Writing), Behaviors (Hallucination, Instruction Following, Refusal Calibration), and Stability (Output Consistency).
Judge Panel: Gemini 2.5 Pro, Claude Opus 4.7, GPT 5.5, and Mistral Large. Each judge scores independently. Final score is the average across all four.
Scoring: Three independent ranks: quality (higher is better), cost (lower is better), speed (faster is better). No composite score — composites hide the tradeoffs that drive model-selection decisions.
Tier Definitions: Leader
Quality ≥ 90.8
Ranks 1–8
Strong
≥ 85.9
Ranks 9–16
Contender
≥ 80.9
Ranks 17–24
Trailing
< 80.9
Ranks 25–32