Production Ingestion Spec — Large Run¶

Budget: $1-5K for medium-sized ingestion (1,000-10,000 papers)¶

1. Smart Routing: OCR+LLM vs VLM¶

Goal¶

Route each paper to the cheapest pipeline that achieves acceptable recall.

Routing Signals (no API calls, pure PDF analysis)¶

Signal	How to compute	Route to VLM if...
Figure/table count	Count `FPDF_PAGEOBJ_IMAGE` objects per page via pypdfium2	>30% of pages have images
Character deficit	Compare actual char count vs expected (~3000 chars/page for text-heavy)	Page has <70% expected chars → likely has figure
Page object density	Pages with many positioned objects but few text chars	High object count + low text = visual content

Implementation¶

def estimate_visual_pages(pdf_path: str) -> float:
    """Estimate fraction of pages with significant visual content."""
    pdf = pdfium.PdfDocument(pdf_path)
    visual_pages = 0
    for page in pdf:
        text = page.get_textpage().get_text_range()
        char_count = len(text)
        expected_chars = 3000  # typical for a full-text page
        img_objects = sum(1 for obj in page.get_objects()
                         if obj.type == pdfium.raw.FPDF_PAGEOBJ_IMAGE)

        # Page is "visual" if: low text + has images, OR significantly below expected chars
        if (char_count < expected_chars * 0.7 and img_objects > 0) or img_objects >= 3:
            visual_pages += 1

    return visual_pages / max(1, len(pdf))

Routing Decision¶

if visual_fraction > 0.3:
    route = VLM (Claude @96dpi, first 7 pages)     → ~$0.095/paper
else:
    route = OCR+LLM (Nova Micro, full text)        → ~$0.050/paper

Expected split¶

Based on our 20-paper analysis: ~65% VLM, ~35% OCR+LLM Blended cost: ~$0.08/paper

2. Reference Handling (No LLM needed)¶

Goal¶

Extract the citation graph without sending references to expensive LLMs.

Approach¶

Grep BibTeX/PMC IDs from PDF text — regex for DOIs, PMIDs, PMC IDs
Skip reference pages for VLM — already implemented in router (saves 1-6 pages/paper)
Build paper-to-paper graph — create edges based on shared citations
Normalize by citation graph — use reference count as a "fame" signal

Implementation¶

import re

DOI_RE = re.compile(r'10\.\d{4,}/[^\s,;]+')
PMID_RE = re.compile(r'PMID:\s*(\d+)', re.IGNORECASE)
PMC_RE = re.compile(r'PMC\d+')

def extract_references(pdf_text: str) -> list[str]:
    """Extract DOIs, PMIDs, PMC IDs from full text."""
    refs = set()
    refs.update(DOI_RE.findall(pdf_text))
    refs.update(f"PMID:{m}" for m in PMID_RE.findall(pdf_text))
    refs.update(PMC_RE.findall(pdf_text))
    return sorted(refs)

Reference Graph Usage¶

Use case	How
Paper-paper edges	Shared references → edge weight
Protein "fame" normalization	Proteins mentioned in many-cited papers get lower novelty score
Orphan disease detection	Diseases with few citing papers flagged as under-studied

3. Section-Level Evaluation¶

Goal¶

Determine which paper sections contain the most entities (are methods/discussion worth processing?).

Approach¶

Use page position as proxy for section (most papers follow IMRaD structure):

Page range (approx)	Section	Expected entity density
1	Abstract/Title	Low — summary only
2-3	Introduction	Medium — mentions key proteins
4-6	Results	High — tables, figures, data
7-8	Discussion	Medium — interprets findings
9+	Methods + References	Low — skip for extraction

Measurement¶

From our page analysis (26 papers, Claude @96dpi):

Pages	Cumulative Recall	Marginal gain
1-3	38.4%	High (intro + some results)
4-5	63.9%	Highest (results/figures)
6-7	76.1%	Good (tables, supplementary results)
8-10	83.5%	Diminishing (discussion, methods)

Conclusion: Process pages 1-7, skip 8+. Methods contribute <7% marginal recall.

4. Logging & Metrics for Large Runs¶

Per-paper logs (capture for analysis)¶

{
  "paper_id": "doi:10.1038/...",
  "route": "vlm",
  "routing_signals": {
    "visual_fraction": 0.45,
    "avg_chars_per_page": 2100,
    "image_objects_total": 12,
    "reference_pages_skipped": 3
  },
  "extraction": {
    "proteins_found": 28,
    "diseases_found": 3,
    "relationships_found": 15,
    "input_tokens": 12500,
    "output_tokens": 800,
    "cost_usd": 0.085,
    "latency_s": 28.4
  },
  "references_extracted": ["10.1038/s41591-024-03164-7", "PMID:38642345"],
  "pages_processed": [0,1,2,3,4,5,6],
  "section_entities": {
    "pages_1_3": 12,
    "pages_4_5": 18,
    "pages_6_7": 8
  }
}

Aggregate metrics to track¶

Metric	Target	Alerts if
Avg protein recall	>75%	<60% on any batch
Cost/paper	<$0.10	>$0.15
Success rate	>95%	<90%
Avg latency	<60s	>120s
Reference extraction rate	>80% papers with refs	<50%
Visual routing %	50-70%	>85% or <30%

5. Cost Projections¶

Scale	OCR-only	VLM-only (7pp)	Hybrid (routed)	With batch API
1,000 papers	$50	$95	$80	$40
5,000 papers	$250	$475	$400	$200
10,000 papers	$500	$950	$800	$400

Additional value captured (no extra LLM cost)¶

Citation graph: ~$0 (regex on already-extracted text)
Section-level entity attribution: ~$0 (page position mapping)
Protein fame scoring: ~$0 (derived from citation graph)

6. Implementation Checklist¶

[ ] Upgrade router with char-deficit heuristic
[ ] Add reference extraction (regex, no LLM)
[ ] Add per-paper structured logging
[ ] Add section-level entity tracking (by page range)
[ ] Build citation graph builder (paper→paper edges from shared refs)
[ ] Add protein/disease fame scoring from citation network
[ ] Implement batch API submission for Bedrock
[ ] Run pilot: 100 papers with full logging
[ ] Validate metrics, tune thresholds
[ ] Scale to 1,000+ papers

7. Future: Dedicated OCR Models (Unlimited-OCR, DeepSeek-OCR)¶

Evaluated¶

Unlimited-OCR (Baidu, June 2026) — a VLM-based document parser built on DeepSeek-OCR. Outputs structured markdown from page images with layout-aware table parsing.

Why NOT to use for most papers¶

Reason	Detail
Double GPU cost	Requires GPU for OCR (self-hosted) + GPU for LLM extraction = 2 inference steps vs our 1
Self-hosting burden	Needs A100/H100, ~$1-4/hr, plus MLOps for model serving
Overkill for digital PDFs	Our papers are typeset digital PDFs — pypdfium2 extracts text at 89% recall for free
Claude VLM already reads tables	When routed to VLM, Claude sees table images and extracts entities directly in one call
Higher DPI requirement	Needs 300 DPI (vs our 96 DPI) — 10x more pixels, more memory, slower

When it WOULD make sense (future consideration)¶

Scanned/handwritten documents — pypdfium2 returns empty text, need actual OCR
Structured table preservation — if KG needs numerical values (p-values, fold changes, assay results) not just entity names
Non-English papers — specialized multilingual OCR may outperform Claude on CJK text
Supplementary data files — Excel-like tables embedded as images in supplementary PDFs

If we test it later¶

Self-host on g5.xlarge (~$1/hr)
Route only table-heavy pages (subset of VLM-routed papers)
Compare: Unlimited-OCR markdown → Nova Micro extraction vs Claude VLM direct
Measure: does structured table text catch entities Claude misses?

Cost comparison¶

Pipeline	GPU hits	Cost/paper	Table quality
pypdfium2 + Nova Micro	1	$0.05	Poor (garbled columns)
Claude VLM Direct	1	$0.07	Good (visual reading)
Unlimited-OCR + LLM	2	$0.15-0.30	Best (structured markdown)

Decision: Not justified for current workload. Revisit if we need structured numerical table data or encounter scanned documents.

8. Run Plan¶

Phase 1: Pilot (100 papers, ~$10)¶

Full logging enabled
Validate routing accuracy
Measure section-level entity distribution
Build initial citation graph

Phase 2: Medium run (1,000 papers, ~$80)¶

Production configuration
Collect fame/novelty scores
Identify under-studied diseases

Phase 3: Full ingestion (5,000-10,000 papers, ~$400-800)¶

Batch API for 50% cost reduction
Complete knowledge graph
Citation network analysis