Skip to content

GraphRAG Eval

Benchmarking and evaluation harness for biomedical knowledge graph extraction. Measures entity extraction accuracy, pipeline cost, and ingestion scalability.

System Overview

graph LR
    subgraph Input
        PDF[PDF Papers]
    end

    subgraph Router["Smart Router (no LLM)"]
        R{Image density?}
    end

    subgraph VLM["VLM Pipeline"]
        RENDER[Render pages @96dpi]
        VLM_CALL[Claude Sonnet VLM]
    end

    subgraph OCR["OCR+LLM Pipeline"]
        PARSE[pypdfium2 text extraction]
        CHUNK[Chunk + Gleaning]
        LLM[Nova Micro LLM]
    end

    subgraph Output
        KG[Knowledge Graph]
        REF[Citation Graph]
    end

    PDF --> R
    R -->|">30% visual"| RENDER --> VLM_CALL --> KG
    R -->|"text-heavy"| PARSE --> CHUNK --> LLM --> KG
    PDF -->|"regex DOI/PMID"| REF

Key Findings

Pipeline Protein Recall Cost/paper Best for
OCR+LLM (Nova Micro) 89% $0.05-0.21 Maximum recall, text-heavy papers
VLM Direct (Claude @96dpi) 84% $0.07-0.15 Image-heavy papers, best precision
Hybrid (auto-routed) 83% $0.08-0.19 Cost-optimized production runs

Quick Start

# Interactive TUI — guided benchmarking, page analysis, batch submission
uv run eval

# Direct CLI
uv run python -m benchmark --mode vlm --models claude-sonnet --dpi 96 -n 20 -r 3 --seed 42

Architecture

graph TD
    subgraph Evaluation["Evaluation Harness"]
        TUI[TUI - uv run eval]
        BENCH[Benchmark Runner]
        PAGE[Page Analysis]
        ROUTE[Router]
        BATCH[Batch Inference]
        REFS[Reference Extractor]
        SECT[Section Analysis]
        LOG[Structured Logger]
    end

    subgraph Infra["Infrastructure"]
        BEDROCK[AWS Bedrock]
        S3[S3 Cache + Batch]
        GOLD[Gold Standard CSV]
    end

    TUI --> BENCH
    TUI --> PAGE
    TUI --> ROUTE
    TUI --> BATCH
    TUI --> REFS
    BENCH --> BEDROCK
    BATCH --> S3
    BENCH --> GOLD
    ROUTE --> SECT

Available Benchmarks

Benchmark What it measures Run
VLM vs OCR+LLM Pipeline comparison: recall, cost, DPI, page cutoff uv run eval (option 1)
Production Spec Scaling plan, routing, reference graph —
Gold Standard Extraction accuracy vs curated labels uv run eval (option 1)
Model Comparison Consistency and garbage rate —

Reports

All interactive reports with charts and per-paper detail:

Tooling

Tool Purpose Cost
uv run eval Interactive TUI for all operations —
benchmark.router PDF routing (VLM vs OCR) Free
benchmark.references Citation graph from PDFs Free
benchmark.section_analysis Section-level entity distribution Free
benchmark.page_analysis Diminishing returns per page API cost
benchmark.batch Bedrock batch inference (50% off) API cost
benchmark.tune_prompt Per-model prompt optimization API cost