Which frontier models see best? An Eagle Eye investigation of vision capabilities

Motivation — why vision matters for sports AI

At Eagle Eye, we see sponsorship measurement and media monitoring as vision problems first: identify a brand mark on a shirt, LED board, or pitch-side asset; decide whether it is fully visible or occluded; and attribute exposure through motion blur and broadcast overlays. Text-only LLMs cannot close that loop. VLMs can answer questions about frames, propose boxes, and read signage, but public academic benchmarks only partly overlap with sports commercial inventory (shirt crest versus LED, virtual LED, or crowd banner). In this investigation, we ask which documented frontier vision systems look strongest on each public axis and where VLMs can strengthen, rather than replace, classical labelling pipelines.

Architecture schematic (qualitative) — not measured EE scores.

Scope and models under study

We studied frontier VLMs against public vision benchmarks and mapped what those results mean for sports measurement. Our scope covers the primary closed APIs in the MMMU-Pro cluster — Anthropic Claude Opus 5.5 / Opus 5 / Opus 4.7, OpenAI GPT-6 Astra (and nearby GPT-5.x rows on the same board), Google Gemini 3.8 / 3.7 Flash and Gemini 3.1 Pro / Gemini 2.5 Pro (video), xAI Grok 4.7 (shipped 21 September 2026; the MMMU-Pro table below still mirrors the BenchLeader row labeled Grok 4.5 on the 26 September snapshot), and historical Grok-1.5V (RealWorldQA) — alongside open / open-weight families with strong published vision cards, including Alibaba Qwen3.8 / Qwen3-VL. We also examine Video-MME leaders (Seed 2.1 Pro, Qwen3.7-Plus, Gemini 2.5 Pro) for match-broadcast relevance. These benchmarks illuminate capability; they are not brand-detection mAP.

Evaluation taxonomy

We organise the evidence into capability axes that matter for sports stacks — a more useful view than a single “best model” scalar:

1. General multimodal reasoning — MMMU-Pro (10-option, vision-hard academic exam).
2. Real-world spatial VQA — RealWorldQA (xAI-origin spatial scenes).
3. OCR / document — DocVQA, OCRBench (signage, contracts, broadcast chyrons).
4. Grounding / referring expression — RefCOCO-avg (box localisation from language).
5. Video understanding — Video-MME (temporal perception over clips).
6. Sports labelling fitness — qualitative: can a VLM propose reliable asset labels / boxes for shirt, LED, virtual LED under broadcast conditions?

Schematic — not measured EE scores.

Methodology

We ran this investigation by synthesising independent public leaderboards — Artificial Analysis MMMU-Pro, the BenchLeader mirror, llm-stats, and published Qwen results — then reviewing how those signals translate to sports-like scenes. The protocol was:

1. Public benchmark synthesis: Artificial Analysis MMMU-Pro and the BenchLeader mirror dated 26 September 2026 provide the primary comparison for closed frontier APIs.
2. Spatial / video signals: we use the llm-stats RealWorldQA and Video-MME tables, keeping their self-reported status visible wherever they appear.
3. Open-weight evidence: Qwen3-VL technical-report figures for DocVQA, OCRBench, and RefCOCO-avg.
4. Qualitative OD review: official Qwen3-VL 2D-grounding cookbook figures on real photos (football pitch points, crowded open-vocab boxes, referring-expression bbox) — published model overlays, not EE mAP and not a sports-inventory bake-off.
5. Labelling analysis: map task types (zero-shot tag, assisted annotation, production detector training) to where VLMs help vs where classical detectors + human-in-the-loop still win.

# Methodology example — eval harness sketch (not an EE production scorer)
# Scores public VQA exact-match and box IoU against a local gold set.
from dataclasses import dataclass
from typing import Iterable

@dataclass
class Box:
    x1: float; y1: float; x2: float; y2: float
    label: str

def iou(a: Box, b: Box) -> float:
    ix1, iy1 = max(a.x1, b.x1), max(a.y1, b.y1)
    ix2, iy2 = min(a.x2, b.x2), min(a.y2, b.y2)
    inter = max(0.0, ix2 - ix1) * max(0.0, iy2 - iy1)
    if inter == 0:
        return 0.0
    area_a = (a.x2 - a.x1) * (a.y2 - a.y1)
    area_b = (b.x2 - b.x1) * (b.y2 - b.y1)
    return inter / (area_a + area_b - inter)

def vqa_exact_match(pred: str, gold: str) -> float:
    return float(pred.strip().lower() == gold.strip().lower())

def score_frame(preds: Iterable[Box], golds: Iterable[Box], thr: float = 0.5) -> dict:
    """Greedy label-matched IoU @ thr — methodology example only."""
    golds = list(golds)
    hits = 0
    used = set()
    for p in preds:
        best_i, best = -1, 0.0
        for i, g in enumerate(golds):
            if i in used or p.label != g.label:
                continue
            s = iou(p, g)
            if s > best:
                best, best_i = s, i
        if best >= thr:
            hits += 1
            used.add(best_i)
    n = max(len(golds), 1)
    return {"precision_proxy": hits / max(len(list(preds)) or 1, 1),
            "recall_proxy": hits / n,
            "matched": hits}

Code block is a methodology example for scoring VQA / IoU locally — it explains the measurement concepts used in this investigation and does not generate the public scores in the tables below.

Results — sourced public tables

Table 1. MMMU-Pro (independent Artificial Analysis runs; BenchLeader mirror, data as of 26 September 2026). Higher is better. Configurations shown are the published thinking / effort variants named on the board.

Artificial Analysis’s own summary page rounds top rows near 88% / 87%; we quote BenchLeader’s one-decimal mirror for the same AA evaluations dated 26 Sep 2026.

Sourced public metrics — not Eagle Eye measured scores.

Table 2. RealWorldQA (spatial VQA). Scores on a 0–1 / percent scale from the llm-stats leaderboard (last updated 26 September 2026). The board marks results as self-reported / unverified; we treat them as a directional signal, not as an independent lab measurement.

Self-reported aggregator metrics — read as directional context.

Table 3. OCR / document / grounding (Qwen3-VL technical report, arXiv:2511.21631) and video (llm-stats Video-MME).

Sourced aggregator metrics — not Eagle Eye measured scores.

Object detection comparison (Qwen3-VL grounding)

Public VQA leaderboards rarely publish sports-asset bounding boxes. Below we show official Qwen3-VL 2D grounding cookbook outputs on real photographs — point grounding on a football pitch, dense open-vocab boxes in a crowded frame, and a tight referring-expression box. Coordinates come from the published Qwen3-VL cookbook (2d_grounding.ipynb); we redraw them here for web delivery. Not measured EE mAP, and not a sports-inventory bake-off — but they replace synthetic schematics with the model's own published grounding demos.

Official Qwen3-VL cookbook grounding figures on real photos (published model coordinates) — qualitative evidence of grounding style; not Eagle Eye measured mAP; not a sports-inventory bake-off.

What these frames are arguing: sports measurement cares about inventory-class precision (shirt vs LED vs virtual LED vs non-inventory distractors), not only “is there a logo somewhere.” Qwen3-VL’s published grounding — points on a pitch, dense open-vocab boxes, referring-expression boxes — shows frontier VLMs can localise well under language prompts. A VLM that answers “yes, Adidas is visible” can still fail commercial valuation if the box is a loose player silhouette or a crowd banner. That is why classical detectors with calibrated IoU thresholds remain central even when VLMs assist annotation.

Can LLMs / VLMs solve labelling?

Honest short answer: they can assist labelling; they do not fully solve production sports labelling alone under rights, PII, and valuation-grade constraints.

• Zero-shot / few-shot tagging — VLMs are strong at open-vocabulary “what brands appear?” questions (related to RealWorldQA / MMMU-style reasoning). Useful for triage and discovery.
• Assisted annotation — propose boxes / polygons for human review. Best when IoU against gold is audited (see harness sketch). Hallucinated boxes and label drift remain real failure modes.
• Production detectors — for shirt / LED / LED-virtual classes at scale, supervised or weakly supervised detectors with frozen taxonomies still win on latency, cost, and auditability.
• Rights & PII — face / minor / crowd identifiable content, and rights-windowed marks, need policy layers that generic VLM APIs do not provide out of the box.

Architecture schematic (qualitative).

Implications for the Eagle Eye stack

For Sponsorlytix-class measurement and monitoring surfaces, these public VLM boards help us identify which APIs are worth piloting for triage, OCR, and assisted annotation. They complement — rather than replace — domain detectors and case-study valuation discipline. On general multimodal reasoning (MMMU-Pro), the late-September 2026 AA cluster is tight among Claude Opus 5.5, GPT-6 Astra, and Gemini 3.8 Flash. On spatial VQA and referring-expression grounding, published Qwen3.x / Qwen3-VL numbers are particularly strong. On video, Seed / Qwen / Gemini 2.5 Pro rows matter for broadcast temporal context.

Where we mention agents such as Koba, that model remains available only inside Eagle Eye dashboards and products, not as a standalone vision API. Internal EE mAP, SoV, and “Vision Arena” scores stay outside this public comparison.

Limitations and future work

• Benchmark ≠ inventory. MMMU-Pro excellence does not imply jersey-mark mAP.
• Self-reported boards. RealWorldQA / Video-MME aggregator rows are useful directional evidence; validate them independently before procurement lock-in.
• Config sensitivity. “Max / high / thinking” effort variants move MMMU-Pro by several points on the same model family.
• Public comparison by design. This article maps external evidence to sports measurement. Future work may add sports-domain protocols and result columns as they mature, without presenting unverified scores as EE measurements.
• Detection figures are Qwen3-VL cookbook demos. Real photos with published model overlays; they show grounding style, not EE mAP or a sports-inventory bake-off.

Safety and responsible use

Vision systems in sports media process faces, minors in crowds, and rights-restricted marks. Labelling automation should preserve human review for valuation claims, refuse disallowed biometric uses, and avoid scraping that violates platform terms. “Sharper vision” is not a license for more aggressive surveillance.

References / sources

1. Artificial Analysis — MMMU-Pro leaderboard: artificialanalysis.ai/evaluations/mmmu-pro
2. BenchLeader — MMMU-Pro mirror (data as of 26 Sep 2026): benchleader.com/benchmarks/aa_mmmu_pro
3. Yue et al. — MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (paper linked from AA).
4. llm-stats — RealWorldQA leaderboard (updated 26 Sep 2026; self-reported): llm-stats.com/benchmarks/realworldqa
5. llm-stats / LLMBoard — Video-MME (Sep 2026).
6. Qwen3-VL Technical Report, arXiv:2511.21631 — DocVQA, OCRBench, RefCOCO-avg, RealWorldQA tables.
7. QwenLM/Qwen3-VL — 2D grounding cookbook (figures in this article): cookbooks/2d_grounding.ipynb
8. BenchmarkList — RefCOCO-avg mirror (Aug 2026 rows citing Qwen reports).
9. xAI — RealWorldQA release context (Grok-1.5V preview era).

Appendix — glossary

MMMU-Pro. Harder multimodal academic exam (10 options; vision-only variants); independently scored by Artificial Analysis.

RealWorldQA. Spatial real-world VQA set originating with xAI’s Grok-1.5V preview.

RefCOCO-avg. Average referring-expression grounding accuracy across RefCOCO / RefCOCO+ / RefCOCOg.

HITL. Human-in-the-loop review before labels enter training or valuation.

Qwen3-VL grounding demos. Official cookbook figures (real photos + published bbox/point overlays) used here as qualitative grounding evidence — not measured EE scores.

Explore the suite →

← All announcements