Ingestion Study Results¶
Run date: 3–5 July 2026
Model: Bedrock Llama 3.3 70B (OCR+LLM mode)
Papers per arm: 500
Total papers processed: 1,931
Total entities extracted: 94,619 nodes · 196,115 relationships
Takeaways¶
3× more entities from proteomics-specific queries
Papers explicitly mentioning Olink, SomaScan, or "plasma protein" yield 76.7 entities/paper vs 25.3 from generic queries. Use targeted vocabulary for bulk ingestion.
Recent broad papers outperform landmark reviews
2025–2026 papers from any journal produce 1.5× more extractable entities than highly-cited top-journal papers. The proteomics data explosion means newer papers contain richer structured data.
Scale projection: 10K papers → 767K nodes at $500
Using the curated strategy with Nova Micro, a full-scale 10K paper ingestion would cost ~$500 and produce approximately 767K nodes and 1.6M relationships.
Entity Density & Volume¶
Summary Table¶
| Study | Arm | Processed | Nodes | Relationships | Nodes/paper | Rels/paper | Errors |
|---|---|---|---|---|---|---|---|
| Neuro | top journals | 492 | 17,792 | 38,495 | 36.2 | 78.2 | 8 |
| Neuro | broad | 490 | 27,274 | 52,175 | 55.7 | 106.5 | 10 |
| CVD | curated | 497 | 38,115 | 81,036 | 76.7 | 163.0 | 3 |
| CVD | random | 452 | 11,438 | 24,409 | 25.3 | 54.0 | 48 |
Key Findings¶
1. Proteomics-specific queries extract 3× more entities¶
CVD curated (76.7 nodes/paper) vs CVD random (25.3 nodes/paper) — a 3.0× improvement.
Papers selected with proteomics-platform vocabulary (Olink, SomaScan, "plasma protein") contain far denser protein–disease relationships than generic CVD + biomarker papers. This makes intuitive sense: papers with actual proteomic assay data list specific protein panels, concentrations, and statistical associations.
2. Broad recent papers outperform top-journal landmarks (with caveats)¶
Neuro broad (55.7 nodes/paper) vs Neuro top journals (36.2 nodes/paper) — 1.5× more entities.
However, this comparison is confounded by a sort-order bias: broad/random arms use Europe PMC's default recency sort, producing a corpus of 2025–2026 papers, while top/curated arms sort by CITED desc, surfacing older 2016–2020 landmark papers. We are comparing two variables simultaneously: query breadth and publication era.
Plausible explanations for the density gap:
- Recent papers (2025–2026) reflect the proteomics data explosion — more studies reporting high-throughput assay results with explicit protein lists
- Top-journal papers are often reviews or framework papers (e.g., NIA-AA diagnostic criteria) that define concepts but contain fewer extractable data points
- Newer papers use standardized nomenclature that LLMs extract more reliably
To isolate the effect of query strategy vs publication date, a follow-up run should sort all arms identically (e.g., all by CITED desc, or date-bucketed random sampling).
3. Error rates correlate with paper availability¶
| Arm | Error rate | Likely cause |
|---|---|---|
| CVD curated | 0.6% | High-quality, well-formatted OA papers |
| Neuro top | 1.6% | Some older papers lack full PMC text |
| Neuro broad | 2.0% | Recent preprints occasionally missing full text |
| CVD random | 9.6% | Many brand-new 2026 papers without full-text deposit yet |
Paper Selection Profiles¶
Citation Distribution¶
| Arm | Min | P25 | Median | P75 | Max |
|---|---|---|---|---|---|
| Neuro top | 172 | 216 | 296 | 460 | 7,837 |
| CVD curated | 204 | 240 | 316 | 477 | 8,699 |
| Neuro broad | 0 | 0 | 0 | 1 | 52 |
| CVD random | 0 | 0 | 0 | 0 | 12 |
Year Distribution¶
Sort-order bias explains the year distribution shape
Why do curated/top lines drop to zero for 2025–2026? Because sort=CITED desc returns the most-cited papers first — papers need years to accumulate citations, so this naturally fills the 500-paper quota with 2016–2022 papers before reaching any recent ones.
Why do broad/random spike at 2025–2026? Because Europe PMC's default sort is relevance-weighted recency — the 500-paper quota fills immediately with the newest papers.
This is not a gap in the database — it's a direct artifact of the sort parameter. The same query with sort=CITED desc would produce a similar 2016–2022 distribution.
Implication for the study: We are comparing two variables simultaneously: query specificity and publication era. The 3× CVD curated advantage is robust (curated papers are older yet still denser), but the 1.5× neuro broad advantage may partly reflect that 2025–2026 papers contain more structured proteomic data as the field matures.
The curated/top arms draw from a 10-year span of established literature. The broad/random arms are almost entirely 2025–2026 papers — fresh science that hasn't accumulated citations yet.
Test Setup¶
Model & Infrastructure¶
| Parameter | Value |
|---|---|
| LLM | us.meta.llama3-3-70b-instruct-v1:0 (Llama 3.3 70B) |
| Service | AWS Bedrock (cross-region inference, us-east-1) |
| Mode | OCR+LLM (text extraction → chunking → LLM entity extraction) |
| Chunking | 512 tokens, 64 token overlap |
| Concurrency | 5 parallel Bedrock calls |
| Entity Resolution | HGNC (~80K aliases) + UniProt (~50K synonyms) |
| Paper source | Europe PMC (full-text OA articles via PMC) |
| Date range | FIRST_PDATE:[2016 TO 2026] |
| Run mode | --dry-run (extract to JSON, no graph writes) |
Query Vocabulary¶
CVD curated (proteomics-specific):
(proteomic OR "plasma protein" OR "circulating biomarker" OR Olink OR SomaScan)
AND (cardiovascular OR "coronary artery" OR "heart failure" OR atherosclerosis)
AND FIRST_PDATE:[2016 TO 2026] AND OPEN_ACCESS:y
CITED desc
CVD random (generic):
(cardiovascular OR cardiac OR atherosclerosis OR heart failure OR myocardial infarction)
AND (protein OR biomarker OR proteomic)
AND FIRST_PDATE:[2016 TO 2026] AND OPEN_ACCESS:y
Neuro top journals (13 high-IF journals + neuro terms):
(JOURNAL:"Nature" OR JOURNAL:"Science" OR JOURNAL:"Cell" OR ... [13 journals])
AND (neurodegeneration OR Alzheimer OR Parkinson OR ALS OR "frontotemporal"
OR "Lewy body" OR tauopathy OR synuclein OR amyloid)
AND (biomarker OR protein OR proteoform OR proteomic
OR "cerebrospinal fluid" OR CSF OR plasma)
AND FIRST_PDATE:[2016 TO 2026] AND OPEN_ACCESS:y
CITED desc
Neuro broad (no journal filter):
(neurodegeneration OR Alzheimer OR Parkinson OR ALS OR "frontotemporal")
AND (biomarker OR protein OR proteoform OR proteomic)
AND FIRST_PDATE:[2016 TO 2026] AND OPEN_ACCESS:y
Implications for KG Construction¶
Combine curated (proteomics-vocabulary, citation-sorted) with broad (recent, uncited) in a 60:40 ratio. This maximizes entity density while capturing emerging biomarkers not yet in the citation canon.
-
Use proteomics-specific queries for maximum entity yield. The
curatedstrategy extracts 3× more per paper. -
Don't over-index on journal prestige. Broad recent papers produce more extractable entities. Top-journal papers are important for citation context but not necessarily the densest data sources.
-
Combine both strategies for the full-scale run:
- Curated arm for high-density extraction (protein panels, assay results)
- Top-journal arm for canonical relationships and review-level context
-
Scale estimates: See Actual Costs section below for projections based on measured $0.04/paper.
Reproducibility¶
# Retry failed arms (uses cached paper lists, auto-detects 0-node arms)
uv run python -m bioingest.ingestion_study --study neuro --papers 500 --dry-run --retry-failed
uv run python -m bioingest.ingestion_study --study cvd --papers 500 --dry-run --retry-failed
# Run from scratch with cached papers (skip Europe PMC fetch)
uv run python -m bioingest.ingestion_study --study cvd --papers 500 --dry-run --use-cached
# Full run to Neptune
uv run python -m bioingest.ingestion_study --study cvd --papers 500 --target neptune --arm curated
Raw Data¶
- Paper lists:
data/extractions/ingestion_study/*.tsv - Summaries:
data/extractions/ingestion_study/summary_*.json - Run log (neuro 500):
data/extractions/ingestion_study/neuro_500_run.log - Study design:
docs/INGESTION_STUDY_DESIGN.md
Actual Costs¶
| Date | Run | Bedrock Cost |
|---|---|---|
| 3 July 2026 | Neuro top (492 papers) + Neuro broad (started, SSO expired at paper ~489 — extracted 0 nodes) | $35 |
| 4 July 2026 | Neuro broad retry (490 papers, ✓) + CVD curated (497 papers, ✓) | $35 |
| 5 July 2026 | Residual / tail calls | $8 |
| Total | $78 |
~1,931 papers successfully extracted at ~$0.04/paper average (Llama 3.3 70B via Bedrock).
Scale Estimates (based on actual $0.04/paper)¶
| Papers | Strategy | Est. Nodes | Est. Rels | Est. Cost |
|---|---|---|---|---|
| 2,000 | Curated | ~153K | ~326K | ~$80 |
| 5,000 | Curated | ~384K | ~815K | ~$200 |
| 10,000 | Mixed (60:40 curated:broad) | ~650K | ~1.4M | ~$400 |
| 10,000 | Curated only | ~767K | ~1.6M | ~$400 |