Production Batch Run — 50K Papers¶
Overview¶
Large-scale entity extraction from open-access proteomics literature using Bedrock Batch inference (50% cost discount).
Architecture¶
flowchart LR
A[Europe PMC\n32 queries] --> B[Paper Manifest\n~35-50K papers]
B --> C{Content Type}
C -->|PMC papers| D[Full-text XML\n~90% of corpus]
C -->|Preprints| E[PDF download\nbioRxiv/medRxiv]
D --> F[Text → LLM\nBedrock Batch]
E --> G[PDF → VLM\nBedrock Batch]
F --> H[Extraction JSON]
G --> H
H --> I[S3 Archive]
H --> J[Neptune Ingest]Query Strategy¶
32 queries in two tiers targeting open-access papers with full text available:
Tier 1: Direct Olink (16 queries, ~18K hits)¶
| Query | Description |
|---|---|
"olink proteomics" |
Direct platform mentions |
"proximity extension assay" proteomics |
PEA technology |
"olink" biomarker |
Biomarker discovery |
"proximity extension" protein |
PEA method papers |
"NPX" "olink" |
Papers reporting NPX values |
"PEA" "olink" |
Abbreviated references |
"olink" "UK Biobank" |
UKB proteomics cohort |
"olink explore" |
Explore 3072 platform |
"olink" "plasma proteomics" |
Plasma-specific |
"olink target" |
Target 96 platform |
"olink" "protein quantification" |
Quantification |
"olink panel" |
Panel references |
"olink inflammation" |
Inflammation panel |
"olink cardiovascular" |
CVD panel |
"olink neurology" |
Neurology panel |
"olink oncology" |
Oncology panel |
Tier 2: Broader Proteomics (16 queries, ~37K hits)¶
| Query | Description |
|---|---|
"proteomic profiling" "inflammation" |
Inflammation proteomics |
"proteomic profiling" "cardiovascular" |
CVD proteomics |
"proteomic profiling" "oncology" |
Cancer proteomics |
"circulating proteins" "disease" |
Circulating biomarkers |
"SomaScan" OR "SomaLogic" |
Competitor platform |
"multiplex immunoassay" protein plasma |
Related technology |
"proteomic" "UK Biobank" |
UKB proteomics (any) |
"plasma proteomics" biomarker |
Plasma biomarkers |
"protein biomarker" "plasma" discovery |
Discovery studies |
"plasma protein" GWAS |
pQTL associations |
"proteomic profiling" "neurodegeneration" |
Neuro proteomics |
"protein quantitative trait loci" pQTL |
Genetic regulation |
"serum proteomics" biomarker |
Serum studies |
"targeted proteomics" "affinity" |
Affinity proteomics |
"protein panel" biomarker disease |
Panel-based studies |
"aptamer" "proteomics" SomaScan |
Aptamer technology |
Yield Estimates¶
| Metric | Count |
|---|---|
| Raw hits (all queries) | ~55,000 |
| After dedup (by PMCID) | ~35,000–45,000 |
| With full-text XML available | ~30,000 |
| With downloadable PDF (preprints) | ~5,000 |
| Total extractable | ~35,000–50,000 |
Two-Track Extraction¶
| Track | Source | Method | Cost/paper | Papers |
|---|---|---|---|---|
| Text | PMC full-text XML | XML → text → claude-sonnet (batch) | $0.033 | ~90% |
| VLM | bioRxiv/medRxiv PDF | PDF → images → claude-sonnet (batch) | $0.034 | ~10% |
Cost Estimates (Bedrock Batch = 50% off)¶
| Papers | Total Cost | Time |
|---|---|---|
| 35,000 | ~$1,155 | 12-24h (async) |
| 50,000 | ~$1,650 | 24-48h (async) |
Artifact Storage¶
All run outputs are persisted for reproducibility and Neptune rebuild:
| Artifact | Format | Storage | Purpose |
|---|---|---|---|
| Paper manifest | JSONL | S3 + Git | Which papers, metadata |
| Run config | JSON | S3 + Git | Reproducibility |
| Full-text XMLs | XML | S3 (cache) | Re-extraction without re-download |
| Batch input | JSONL | S3 (ephemeral) | Bedrock input, can delete after |
| Extraction results | JSONL | S3 + local | Neptune rebuild backup |
| Run logs | text | S3 | Debugging |
| Stats summary | JSON | S3 + Git | Quick overview |
S3 Layout¶
s3://bioingest-datalake-002356212513/
└── graphrag-eval/
└── runs/
├── run_20260712_143000/
│ ├── manifest.json # Run manifest (timing, config, artifact index)
│ ├── config.json # Exact config used
│ ├── extractions.jsonl # THE KEY ARTIFACT — all entities extracted
│ ├── paper_index.jsonl # Paper metadata
│ ├── batch_jobs.json # Bedrock job ARNs
│ ├── stats.json # Summary stats
│ └── run.log # Execution log
└── cache/
└── xml/ # Full-text XML cache (optional sync)
Extraction JSON Schema¶
Each line in extractions.jsonl:
{
"doc_id": "PMC10567551",
"status": "success",
"raw_response": "...",
"parsed": {
"proteins": ["IL-6", "TNF", "CCL2", ...],
"diseases": ["rheumatoid arthritis", ...],
"relationships": [
{"protein": "IL-6", "disease": "rheumatoid arthritis", "relationship": "biomarker"}
]
},
"input_tokens": 14520,
"output_tokens": 1893
}
This format allows:
1. Full Neptune rebuild from parsed.proteins + parsed.diseases + parsed.relationships
2. Re-parsing from raw_response if schema changes
3. Cost auditing from token counts
Running¶
Interactive (TUI)¶
CLI¶
# Step 1: Source papers
uv run python -m benchmark.sourcer -n 50000
# Step 2: Estimate cost
uv run python -m benchmark.prod_batch --estimate
# Step 3: Prepare + submit
uv run python -m benchmark.prod_batch --mode auto --submit
# Step 4: Monitor
uv run python -m benchmark.prod_batch --poll benchmark/results/prod_batch/*_jobs.json
# Step 5: Collect results
uv run python -m benchmark.prod_batch --collect benchmark/results/prod_batch/*_jobs.json
Recall Justification¶
From the benchmark (20 papers × 5 runs, gold standard comparison):
| Mode | Model | Recall | Precision | F1 |
|---|---|---|---|---|
| VLM | claude-sonnet | 83.9% | 26.7% | 37.7% |
| OCR+LLM | claude-sonnet | 90.2% | 24.2% | 35.7% |
| OCR+LLM | llama3-3-70b | 90.7% | 22.5% | 34.3% |
Text-based extraction (OCR+LLM) achieves 90%+ recall — using XML full-text is equivalent to the OCR+LLM path (same text quality, no PDF parsing errors). This is the primary mode for the production run.