Skip to content

Incremental Updates

Weekly maintenance workflow for keeping the knowledge graph current with new PubMed publications. Optimized for <15 minute completion.

Quick Start

# Default: last 7 days, 100 abstracts max, local LLM
uv run python scripts/incremental_pubmed_update.py \
  --search-term "cardiovascular disease protein biomarker" \
  --database olink1

# Custom time window
uv run python scripts/incremental_pubmed_update.py \
  --search-term "kidney disease biomarker" --days 14 --database olink1

# Production (AWS Bedrock)
uv run python scripts/incremental_pubmed_update.py \
  --search-term "cancer protein marker" --service bedrock --database olink1

What Happens

The script (scripts/incremental_pubmed_update.py) runs a 5-step pipeline:

Step 1: Fetch new abstracts     → Date-filtered PubMed query + PMID deduplication
Step 2: Extract entities        → LLM-based extraction with token chunking (512 tokens)
Step 3: Consolidate entities    → UniProt ID matching against existing nodes
Step 4: Consolidate relations   → Merge edges for new PMIDs, preserve evidence
Step 5: Generate embeddings     → Only for nodes where embedding IS NULL

Timing Breakdown

Step Duration Details
Fetch new abstracts 2-3 min PubMed Entrez API with date filter
Extract entities 5-8 min LLM calls per chunk (most expensive step)
Consolidate entities 1-2 min Fast UniProt ID matching
Consolidate relationships 1-2 min Incremental merge for new PMIDs only
Generate embeddings 2-3 min sentence-transformers on new content
Total 11-18 min Target: <15 min for 100 abstracts

CLI Options

Flag Default Description
--search-term / -t required PubMed search query
--days / -d 7 Number of days to look back
--max-results / -n 100 Maximum abstracts to process
--service / -s local LLM service (local, bedrock, sagemaker-llama3, openai)
--database / -db olink3 Neo4j database name

Date Filtering

The script builds a PubMed date publication filter internally:

"cardiovascular disease protein biomarker" AND ("2026/07/13"[Date - Publication] : "2026/07/20"[Date - Publication])

No need to pass dates manually — the --days flag handles it.

PMID Deduplication

Already-processed PMIDs are skipped automatically. The KGPipeline tracks processed PMIDs in Neo4j and marks them after successful ingestion. This means:

  • Running the script twice with the same parameters is safe (idempotent)
  • No duplicate abstracts will be processed
  • The script reports "No new abstracts found. Database is up to date." if nothing is new

Weekly Cron Job

# Every Monday at 2am
0 2 * * 1 cd /path/to/olink_rag && uv run python scripts/incremental_pubmed_update.py \
  --search-term "protein disease biomarker" --days 7 \
  --database olink1 --service bedrock >> logs/weekly_update.log 2>&1

For multiple search terms, use separate cron entries or a wrapper script:

#!/bin/bash
# scripts/weekly_update_all.sh
set -e
TERMS=(
  "cardiovascular disease protein biomarker"
  "cancer protein biomarker"
  "kidney disease protein biomarker"
  "neurological disease protein biomarker"
)
for term in "${TERMS[@]}"; do
  echo "$(date): Updating: $term"
  uv run python scripts/incremental_pubmed_update.py \
    --search-term "$term" --days 7 --database olink1 --service bedrock
done

bioRxiv Incremental Updates

The --date-from/--date-to flags in ingest_main.py are bioRxiv-specific:

uv run python ingest_main.py \
  --source biorxiv --search-term "protein biomarker" \
  --date-from $(date -v-7d +%Y-%m-%d) \
  --date-to $(date +%Y-%m-%d) \
  --database olink1 --service bedrock

# Then consolidate
uv run python ingest_main.py --consolidate-relationships --database olink1
uv run python ingest_main.py --add-graph-embeddings --database olink1 --service bedrock

Output Statistics

The script logs a summary on completion:

============================================================
INCREMENTAL UPDATE COMPLETE
============================================================
Search term: cardiovascular disease protein biomarker
Time window: Last 7 days
New abstracts: 47
New entities: 312
New relationships: 189
Consolidated entities: 28
Consolidated relationships: 45
Total duration: 683.2s (11.4 min)
============================================================

Monitoring

After each update cycle, verify the KG health:

# Check cache stats (should invalidate stale entries)
curl "http://localhost:8000/v1/cache/stats" -H "Authorization: Bearer $API_AUTH_TOKEN"

# Invalidate cache after graph updates
curl -X POST "http://localhost:8000/v1/cache/invalidate?database=olink1" \
  -H "Authorization: Bearer $API_AUTH_TOKEN"

# Check incremental update metrics
curl "http://localhost:8000/v1/metrics/incremental-updates" -H "Authorization: Bearer $API_AUTH_TOKEN"

Troubleshooting

Symptom Cause Fix
"No new abstracts found" Already up-to-date or too narrow search Increase --days or broaden search term
Slow extraction (>20 min) local LLM too slow for 100 abstracts Switch to --service bedrock
Entity consolidation low New entities don't match existing UniProt IDs Run full consolidation: uv run python -m pipeline.processors.entity_resolver --database olink1 --operation full
Embeddings fail Embedding model not loaded Check sentence-transformers is installed and model downloads
Script crashes mid-run Network/LLM timeout Re-run — PMID deduplication means it picks up where it left off