VLM vs OCR+LLM Benchmark¶
Comparison of three extraction pipeline architectures for biomedical knowledge graph extraction.
Date: 2026-06-12 | Papers: 20 | Gold Standard: Gary's curated protein/disease labels | DPI: 200
Interactive Reports (charts + per-paper detail)¶
- Overnight Benchmark (DPI + Models + Pages) →
- Combined: OCR+LLM vs VLM vs VLM+LLM →
- VLM Direct Report →
- OCR+LLM Report →
- VLM+LLM Report →
- Original Parser x Model Report →
Pipeline Modes¶
| Mode | Pipeline | Description |
|---|---|---|
| OCR+LLM | PDF → PyMuPDF → text → LLM | Traditional parser-based text extraction |
| VLM Direct | PDF → page images → VLM | Vision model extracts entities directly from rendered pages |
| VLM+LLM | PDF → page images → VLM (transcribe) → text → LLM | VLM as "smart OCR", LLM for extraction |
Summary Results¶
Best model per mode¶
| Mode | Best Model | Protein F1 | Disease F1 | Reproducibility | Cost (20 papers) | Success |
|---|---|---|---|---|---|---|
| VLM Direct | Claude Sonnet 4.6 | 36.6% | 18.8% | 86.2% | $8.15 | 100% |
| OCR+LLM (pymupdf) | Llama 3.3 70B | 36.2% | 10.8% | 97.8% | $15.36 | 100% |
| OCR+LLM (pymupdf) | Claude Sonnet 4.6 | 33.9% | 10.5% | 95.8% | $26.62 | 100% |
| VLM+LLM (via Nova Pro) | Claude Sonnet 4.6 | 25.9% | 14.2% | 56.9% | $12.32 | 45% |
| VLM+LLM (via Nova Pro) | Llama 3.3 70B | 24.7% | 12.5% | 59.2% | $16.53 | 100% |
Key Finding
VLM Direct (Claude Sonnet) matches OCR+LLM on protein F1 (36.6% vs 36.2%) and outperforms on disease F1 by 73% (18.8% vs 10.8%) — at half the cost ($8.15 vs $15.36). Vision models reading PDF pages directly find more diseases than text parsers + LLMs.
VLM+LLM Improved
After page batching fix (3 pages per VLM call), VLM+LLM now achieves 100% success with Llama 3.3 70B (24.7% P-F1). Still underperforms direct approaches but is cost-effective and shows potential with prompt tuning.
Absolute F1 is Low Across All Modes
All modes score under 40% protein F1. This reflects the strictness of the gold standard (exact-match against Gary's curated labels) and the challenge of full-paper extraction. The relative comparison between modes is the important signal.
Recall Analysis (200 DPI)¶
Recall is the primary metric of interest — how many gold-standard entities does each pipeline find?
Protein Recall¶
| Mode | Model | Protein Recall | Disease Recall | Notes |
|---|---|---|---|---|
| OCR+LLM | Nova Pro | 89.1% | 75.0% | Highest protein recall — aggressive chunking + gleaning |
| OCR+LLM | Claude Sonnet | 88.9% | 75.0% | Near-identical recall to Nova Pro |
| OCR+LLM | Llama 3.3 70B | 88.9% | 72.5% | Best F1 due to better precision |
| VLM Direct | Claude Sonnet | 84.6% | 72.5% | Close to OCR+LLM, much better precision |
| VLM+LLM | Claude Sonnet | 64.1% | 77.8% | Best disease recall overall |
| VLM+LLM | Llama 3.3 70B | 61.2% | 62.5% | Signal degrades in transcription handoff |
| VLM Direct | Nova Pro | 21.9% | 30.0% | Poor — model struggles with extraction prompt |
Key Recall Findings¶
- OCR+LLM has highest protein recall (~89%) because text chunking + gleaning is exhaustive
- VLM Direct (Claude) is close at 84.6% protein recall — only 4.5% gap vs OCR+LLM
- VLM+LLM loses recall — the transcription step drops protein recall to ~61-64%
- Disease recall is more even across modes (72-78%) — diseases are mentioned more prominently
- 100 DPI test needed — will VLM recall hold at lower resolution?
Detailed Results per Mode¶
VLM Direct (20 papers x 5 runs, seed 42)¶
| Model | P-Prec | P-Rec | P-F1 | D-Prec | D-Rec | D-F1 | Repro | Cost | OK% |
|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 4.6 | — | — | 36.6% | — | — | 18.8% | 86.2% | $8.15 | 100% |
| Nova Pro | — | — | 15.8% | — | — | 14.9% | 85.4% | $36.04 | 100% |
OCR+LLM -- PyMuPDF parser (20 papers x 5 runs, seed 42)¶
| Model | P-Prec | P-Rec | P-F1 | D-Prec | D-Rec | D-F1 | Repro | Cost | OK% |
|---|---|---|---|---|---|---|---|---|---|
| Llama 3.3 70B | — | — | 36.2% | — | — | 10.8% | 97.8% | $15.36 | 100% |
| Claude Sonnet 4.6 | — | — | 33.9% | — | — | 10.5% | 95.8% | $26.62 | 100% |
| Nova Pro | — | — | 27.6% | — | — | 11.5% | 88.2% | $53.31 | 100% |
VLM+LLM via Nova Pro (20 papers x 5 runs, seed 42)¶
| Model (extractor) | P-F1 | D-F1 | Repro | Cost | OK% |
|---|---|---|---|---|---|
| Claude Sonnet 4.6 | 25.9% | 14.2% | 56.9% | $12.32 | 45% |
| Llama 3.3 70B | 24.7% | 12.5% | 59.2% | $16.53 | 100% |
| Mistral Large 3 | 0.0% | 0.0% | 0.0% | $0.00 | 0% |
Mistral Large: API access not available or model errors. Claude: some timeouts on transcription step.
Key Takeaways¶
- VLM Direct is the best architecture for disease extraction — 73% better disease F1, at half the cost of OCR+LLM
- VLM Direct and OCR+LLM are comparable on protein F1 — Claude Sonnet VLM (36.6%) vs Llama 70B OCR+LLM (36.2%)
- OCR+LLM has highest reproducibility — Llama 70B achieves 97.8% (near-deterministic at temperature 0)
- VLM+LLM is viable after page batching fix — Llama 70B achieves 100% success, 24.7% P-F1
- Claude Sonnet is the most versatile model — performs well in all three modes
Reproduce¶
cd graphrag_eval
aws sso login # ensure fresh credentials
# OCR+LLM (pymupdf, ~15 min)
uv run python -m benchmark --mode ocr+llm --parser pymupdf \
--models nova-pro,claude-sonnet,llama3-3-70b -n 20 -r 5 --seed 42
# VLM Direct (~20 min, concurrency auto-capped to 3)
uv run python -m benchmark --mode vlm \
--models nova-pro,claude-sonnet -n 20 -r 5 --seed 42
# VLM+LLM (~25 min)
uv run python -m benchmark --mode vlm+llm --vlm-model nova-pro \
--models llama3-3-70b,claude-sonnet,mistral-large -n 20 -r 5 --seed 42
Relation to OCRBench v2 / MultimodalOCR¶
OCRBench v2 and MultimodalOCR are the reference benchmarks for evaluating VLM OCR capabilities. Here we assess what can and cannot be adapted to our biomedical extraction task.
What CAN be adapted (without re-running)¶
Difficulty stratification by document complexity. OCRBench separates easy vs hard samples. We can do the same using metadata already in our results:
| Difficulty | Criteria | Papers | Hypothesis |
|---|---|---|---|
| Easy | ≤12 pages, ≤4 image-heavy pages | 4 papers | VLM and OCR+LLM should perform similarly — mostly text |
| Hard | >20 pages, >10 image-heavy pages | 5 papers | VLM should win big — tables/figures contain entities that text parsers miss |
| Medium | Everything else | 11 papers | Mixed results |
Our dataset averages 22 pages with 8.8 image-heavy pages per paper. The papers with the most figures/tables (microglia, proteomics, multi-omics) are exactly where VLM's advantage should be strongest. This stratification can be computed from existing run data by cross-referencing per-paper F1 scores against page/image counts.
Task decomposition (partial). OCRBench breaks OCR into sub-tasks (recognition, extraction, reasoning). We can approximate this by separately scoring:
- Inline text entities — proteins/diseases mentioned in running text (both modes should find these)
- Table/figure entities — proteins in tables, heatmaps, volcano plots (VLM advantage)
- Reference-list entities — names in citations (noise — both modes should ignore these)
This requires no re-running — just manual tagging of which gold-standard entities appear in tables vs text for a subset of papers.
What CANNOT be adapted¶
| OCRBench Feature | Why it doesn't fit | Rationale |
|---|---|---|
| Character-level accuracy metrics | We extract entities, not raw text | Our task is semantic (find "IL-6") not character-accurate OCR. A model that OCRs "IL-6" perfectly but doesn't recognize it as a protein scores 0 in our benchmark. |
| Bounding box / localization tasks | No spatial ground truth | Our gold standard has entity names only, not coordinates. We don't care where on the page "GDF15" appears, only that it's extracted. |
| Scene text / handwriting tasks | Biomedical PDFs are digital documents | OCRBench tests street signs, handwritten forms, receipts. Our PDFs are typeset — the OCR challenge is tables/figures, not degraded text. |
| Multilingual evaluation | All papers are English | OCRBench v2 covers 9+ languages. Our corpus is exclusively English biomedical literature. |
| VQA (Visual Question Answering) | Wrong task framing | OCRBench asks "What text is in this region?" We ask "What proteins are discussed in this paper?" — a semantic extraction task, not a reading comprehension task. |
| Public/private split for leakage prevention | Not applicable | Our models aren't fine-tuned on these papers. Leakage is only a concern for trained models. We use general-purpose LLMs/VLMs at temperature 0. |
| Standardized scoring (0-1000 scale) | Incompatible with F1 metrics | OCRBench uses a point system across diverse tasks. Our single-task F1 is more interpretable for stakeholders comparing pipelines. |
Bottom line¶
OCRBench measures "can the model read?" — character recognition, text localization, visual reasoning about text in images. Our benchmark measures "can the model understand and extract domain knowledge?" — a fundamentally different question. A model scoring 90% on OCRBench might still score 20% on our benchmark if it reads text perfectly but can't distinguish protein names from English words.
The one transferable insight is difficulty stratification — and we can apply it retroactively to existing results by correlating per-paper F1 with document complexity (page count, figure density).