GraphRAG Eval¶

Benchmarking and evaluation harness for biomedical knowledge graph extraction. Measures entity extraction accuracy, pipeline cost, and ingestion scalability.

System Overview¶

graph LR
    subgraph Input
        PDF[PDF Papers]
    end

    subgraph Router["Smart Router (no LLM)"]
        R{Image density?}
    end

    subgraph VLM["VLM Pipeline"]
        RENDER[Render pages @96dpi]
        VLM_CALL[Claude Sonnet VLM]
    end

    subgraph OCR["OCR+LLM Pipeline"]
        PARSE[pypdfium2 text extraction]
        CHUNK[Chunk + Gleaning]
        LLM[Nova Micro LLM]
    end

    subgraph Output
        KG[Knowledge Graph]
        REF[Citation Graph]
    end

    PDF --> R
    R -->|">30% visual"| RENDER --> VLM_CALL --> KG
    R -->|"text-heavy"| PARSE --> CHUNK --> LLM --> KG
    PDF -->|"regex DOI/PMID"| REF

Key Findings¶

Pipeline	Protein Recall	Cost/paper	Best for
OCR+LLM (Nova Micro)	89%	$0.05-0.21	Maximum recall, text-heavy papers
VLM Direct (Claude @96dpi)	84%	$0.07-0.15	Image-heavy papers, best precision
Hybrid (auto-routed)	83%	$0.08-0.19	Cost-optimized production runs

Quick Start¶

# Interactive TUI — guided benchmarking, page analysis, batch submission
uv run eval

# Direct CLI
uv run python -m benchmark --mode vlm --models claude-sonnet --dpi 96 -n 20 -r 3 --seed 42

Architecture¶

graph TD
    subgraph Evaluation["Evaluation Harness"]
        TUI[TUI - uv run eval]
        BENCH[Benchmark Runner]
        PAGE[Page Analysis]
        ROUTE[Router]
        BATCH[Batch Inference]
        REFS[Reference Extractor]
        SECT[Section Analysis]
        LOG[Structured Logger]
    end

    subgraph Infra["Infrastructure"]
        BEDROCK[AWS Bedrock]
        S3[S3 Cache + Batch]
        GOLD[Gold Standard CSV]
    end

    TUI --> BENCH
    TUI --> PAGE
    TUI --> ROUTE
    TUI --> BATCH
    TUI --> REFS
    BENCH --> BEDROCK
    BATCH --> S3
    BENCH --> GOLD
    ROUTE --> SECT

Available Benchmarks¶

Benchmark	What it measures	Run
VLM vs OCR+LLM	Pipeline comparison: recall, cost, DPI, page cutoff	`uv run eval` (option 1)
Production Spec	Scaling plan, routing, reference graph	—
Gold Standard	Extraction accuracy vs curated labels	`uv run eval` (option 1)
Model Comparison	Consistency and garbage rate	—

Reports¶

All interactive reports with charts and per-paper detail:

Production Validation — compiled results across all runs
Hybrid Routing Pilot — auto VLM/OCR routing
DPI + Pages Eval — DPI floor and page cutoff analysis
Scaling Analysis — cost projections at 1K-100K papers
Combined Mode Comparison — OCR vs VLM vs VLM+LLM

Tooling¶

Tool	Purpose	Cost
`uv run eval`	Interactive TUI for all operations	—
`benchmark.router`	PDF routing (VLM vs OCR)	Free
`benchmark.references`	Citation graph from PDFs	Free
`benchmark.section_analysis`	Section-level entity distribution	Free
`benchmark.page_analysis`	Diminishing returns per page	API cost
`benchmark.batch`	Bedrock batch inference (50% off)	API cost
`benchmark.tune_prompt`	Per-model prompt optimization	API cost