VLM Benchmark — DPI Comparison
20 papers, 5 runs/paper, seed 42 — comparing 100 vs 150 vs 200 DPI across OCR+LLM, VLM Direct, VLM+LLM
DPI has minimal impact on the best VLM (Claude Sonnet)
Claude VLM Direct recall: 84.1% @100dpi, 85.4% @150dpi, 84.6% @200dpi. The difference is within noise. Lower DPI saves cost ($7.6 vs $8.4) with no quality loss. Other models (Nova Pro, Nova Lite) perform poorly regardless of DPI.
OCR+LLM still wins on raw protein recall (~89%) but at much higher cost
All OCR+LLM models achieve ~89% protein recall regardless of model choice. Best cost-performance: Nova Micro ($5.0) or Llama 70B ($15.5). VLM Direct (Claude @100dpi) achieves 84% at only $7.6.
DPI Impact on VLM Direct (Claude Sonnet)
Best Model per Mode (Protein Recall)
Full Results — VLM Direct by DPI
| DPI | Model | P-Recall | D-Recall | P-F1 | Cost | OK% |
|---|
| 100 | Claude Sonnet 4.6 | 84.1% | 72.5% | 35.4% | $7.60 | 100% |
| 150 | Claude Sonnet 4.6 | 85.4% | 72.5% | 36.8% | $8.30 | 100% |
| 200 | Claude Sonnet 4.6 | 84.6% | 72.5% | 36.6% | $8.15 | 100% |
| 100 | Nova Pro | 25.1% | 35.0% | 18.2% | $36.50 | 100% |
| 150 | Nova Pro | 23.3% | 30.0% | 12.5% | $34.00 | 100% |
| 200 | Nova Pro | 21.9% | 30.0% | 15.8% | $36.04 | 100% |
Full Results — VLM+LLM by DPI (best models only)
| DPI | Extractor | P-Recall | D-Recall | P-F1 | Cost | OK% |
|---|
| 100 | Nova Lite | 66.2% | 70.0% | 24.4% | $33.50 | 100% |
| 100 | Nova Micro | 62.0% | 67.5% | 23.9% | $5.50 | 100% |
| 100 | Llama 3.3 70B | 60.7% | 60.0% | 26.7% | $16.10 | 100% |
| 150 | Nova Pro | 67.4% | 60.0% | 20.7% | $64.00 | 100% |
| 150 | Nova Micro | 66.2% | 65.0% | 24.0% | $5.40 | 100% |
| 150 | Llama 3.3 70B | 57.4% | 57.5% | 24.4% | $16.20 | 100% |
Full Results — OCR+LLM (DPI-independent)
| Model | P-Recall | D-Recall | P-F1 | Cost | OK% |
|---|
| Llama 3.1 8B | 89.4% | 80.0% | 24.3% | $18.20 | 100% |
| Llama 3.3 70B | 88.9% | 77.5% | 35.2% | $15.50 | 100% |
| Nova Micro | 89.2% | 77.5% | 29.9% | $5.00 | 100% |
| Claude Sonnet 4.6 | 88.9% | 80.0% | 33.3% | $26.90 | 100% |
| Nova Lite | 89.1% | 80.0% | 29.7% | $31.20 | 100% |
| Nova Pro | 89.1% | 80.0% | 25.1% | $54.40 | 100% |
| Mistral Large 3 | 88.9% | 80.0% | 26.9% | $25.40 | 100% |
Conclusions
Recommended configuration: VLM Direct, Claude Sonnet, 100 DPI
84% protein recall, 73% disease recall, $7.60 for 20 papers. Reducing DPI from 200 to 100 saves 10% cost with no meaningful recall loss. If maximum recall is critical, OCR+LLM with Nova Micro ($5.00, 89% recall) is cheapest but has lower precision.
Model Viability for VLM Direct
Nova Lite and Nova Micro are not viable for VLM Direct extraction
Nova Lite degenerates into repetition loops when processing document images at temperature 0, generating the same tokens cyclically (e.g., "IL-6, TNF, IL-10, IL-6, TNF, IL-10...") until maxTokens is hit. This produces invalid JSON and 0% recall regardless of prompt design, DPI, or temperature settings. This is a model capacity limitation, not a prompt issue.
| Model | VLM Direct Viable? | Best Use | Notes |
|---|
| Claude Sonnet 4.6 | Yes — 84% recall | VLM Direct | Best quality, handles complex prompts |
| Nova Pro | Marginal — 25% recall | VLM+LLM transcriber | Low recall even with tuned prompts; works as transcriber |
| Nova Lite | No — repetition loops | VLM+LLM extractor | Degenerates on image tasks; OK for text extraction |
| Nova Micro | N/A — text only | OCR+LLM extractor | Cheapest LLM ($5/20 papers) at 89% recall |
| Llama 3.2 90B Vision | No — API errors | — | Not accessible on Bedrock in current account |
Cheapest production options ranked by recall
| Rank | Pipeline | P-Recall | Cost/20 papers | Cost/paper |
|---|
| 1 | OCR+LLM / Nova Micro | 89.2% | $5.00 | $0.25 |
| 2 | VLM Direct / Claude @100dpi | 84.1% | $7.60 | $0.38 |
| 3 | OCR+LLM / Llama 3.3 70B | 88.9% | $15.50 | $0.78 |
| 4 | VLM+LLM / Nova Micro @100dpi | 62.0% | $5.50 | $0.28 |
Methodology
| Mode | Pipeline |
|---|
| OCR+LLM | PDF - PyMuPDF text - chunk 512 tokens - LLM extraction - gleaning pass |
| VLM Direct | PDF - render pages as PNG (variable DPI, max 10pp) - VLM extracts entities from images |
| VLM+LLM | PDF - render pages - VLM transcribes to markdown (3pp/batch) - LLM extraction from text |