Skip to content

Parallel Ingestion

Run PubMed, PMC, and bioRxiv ingestion in parallel for bulk KG building. Each source gets its own subprocess with independent Neo4j connections and LLM instances. Neo4j handles concurrent writes safely via its transaction model.

Quick Start

# Dry run — show what would be launched without running
uv run python parallel_ingest.py \
  -q data/queries_bulk.txt \
  --biorxiv-queries-file data/queries_biorxiv.txt \
  -d olink1 -s bedrock --dry-run

# Full parallel run (overnight)
uv run python parallel_ingest.py \
  -q data/queries_bulk.txt \
  --biorxiv-queries-file data/queries_biorxiv.txt \
  -d olink1 -s bedrock \
  --skip-node-labeling \
  --pubmed-max 500 --pmc-max 200 --biorxiv-max 100

How It Works

parallel_ingest.py
├── Parses queries file (one MeSH query per line)
├── Builds ingest_main.py commands for each enabled source
├── Launches all sources as async subprocesses
├── Tails each log in real-time (5s interval)
├── Waits for all to complete
└── Prints summary: ✅/❌ per source + wall-clock time

Logs are written to logs/parallel_YYYYMMDD_HHMMSS/{source}.log.

All Options

Flag Default Description
--queries-file / -q required Queries file (used for PubMed and PMC)
--biorxiv-queries-file same as -q Separate queries for bioRxiv (simple keywords)
--database / -d olink3 Neo4j database name
--service / -s local LLM service (local, bedrock, sagemaker-llama3)
--pubmed-max 500 Max results per query for PubMed
--pmc-max 200 Max results per query for PMC
--biorxiv-max 100 Max results per query for bioRxiv
--skip-pubmed off Skip PubMed source
--skip-pmc off Skip PMC source
--skip-biorxiv off Skip bioRxiv source
--skip-node-labeling off Skip post-ingestion labeling (saves hours)
--enable-consolidation off Run relationship consolidation during ingestion
--force / -f off Force re-ingestion of already-processed PMIDs
--chunk-size 2000 Chunk size for full-text sources (words)
--chunk-overlap 200 Chunk overlap (words)
--pubmed-mass-scraper off Use mass scraper for PubMed (millions of abstracts)
--batch-size 500 Mass scraper batch size
--max-concurrent-batches 3 Mass scraper concurrency
--dry-run off Show commands without running

Query Files

data/queries_bulk.txt (74 lines, ~31 active queries)

MeSH-style queries for PubMed and PMC. One per line, supports # comments:

# Cardiovascular
"cardiovascular disease"[MeSH] AND "protein biomarker"
"heart failure"[MeSH] AND "proteomics"

# Oncology
"breast cancer"[MeSH] AND "serum biomarker"

data/queries_biorxiv.txt (37 lines)

Simple keyword terms — bioRxiv API does substring matching on title/abstract:

protein biomarker cardiovascular
proteomics cancer serum

Performance Tips

  1. Use bedrock or sagemaker-llama3 for bulk runs — local (Ollama) is too slow for hundreds of articles.
  2. Always --skip-node-labeling during bulk ingestion — saves hours. Run labeling separately on a schedule (see Operations).
  3. PubMed is fastest (abstracts only). PMC and bioRxiv are slower due to full-text chunking.
  4. Overnight runs through AWS SSM: use caffeinate -s bash scripts/tunnel_keepalive.sh to prevent tunnel drops.
  5. Monitor progress: tail the per-source logs in logs/parallel_*/.

Post-Ingestion Steps

After parallel ingestion completes, run these sequentially:

# 1. Entity consolidation
uv run python -m pipeline.processors.entity_resolver \
  --database olink1 --operation full

# 2. Relationship consolidation
uv run python ingest_main.py --consolidate-relationships --database olink1

# 3. Node labeling (if skipped during ingestion)
uv run python ingest_main.py --label-nodes -d olink1

# 4. Vector embeddings
uv run python ingest_main.py --add-graph-embeddings --database olink1 --service bedrock

Typical Timings

Source Queries Max/query ~Duration ~Articles
PubMed 31 500 2-4 hours ~5,000
PMC 31 200 4-8 hours ~2,000
bioRxiv 15 100 2-3 hours ~500
Total (parallel) — — 4-8 hours ~7,500

Wall-clock time equals the slowest source (they run simultaneously).

Troubleshooting

Symptom Cause Fix
All sources fail immediately Neo4j not running Start Neo4j: docker compose --profile db up -d
bioRxiv returns 0 results Keywords too long Use 2-3 word terms in bioRxiv queries file
Process killed overnight SSH/SSM tunnel dropped Use caffeinate -s + tunnel_keepalive.sh
Neo4j OOM Too many concurrent writes Reduce --pubmed-max or skip one source