Getting Started

From install to first query in minutes. Interactive TUI or scriptable CLI — your choice.

# Install
$ git clone https://github.com/Olink-Proteomics/bioingest.git && cd bioingest
$ uv sync

# Launch interactive pipeline
$ uv run bioingest

    ____  _       ____                      __
   / __ )(_)___  /  _/___  ____ ____  _____/ /_
  / __  / / __ \ / // __ \/ __ `/ _ \/ ___/ __/
 / /_/ / / /_/ // // / / / /_/ /  __(__  ) /_
/_____/_/\____/___/_/ /_/\__, /\___/____/\__/
                        /____/

  ① Fetch Data          download, scrape, or pull raw data
  ② Transform / Edit    run scripts, preview files, normalize
  ③ Publish to S3       convert to Parquet, upload, create tables
  ④ Query / Explore     Athena SQL, Jupyter, overlap matrix
  ⑤ KG Ingestion        build knowledge graph from PDFs
  ⑥ Publication Timeline bioRxiv→PMC preprint tracking
  ⑦ Pipeline Status     overview of local data + AWS
  ⑧ Eval                extraction benchmark (graphrag_eval)

---

Install

git clone https://github.com/Olink-Proteomics/bioingest.git
cd bioingest
uv sync                        # core (download + publish)
uv sync --extra pipeline       # + KG construction (Neo4j, LLM, PDF)
uv sync --extra scrape         # + web scraping (Firecrawl)
uv sync --extra notebook       # + Jupyter exploration

Command	What it does
uv run bioingest	Interactive TUI — guided workflow with menus
uv run bioingest-cli	Non-interactive CLI — scriptable commands

---

The Pipeline

1

### Fetch Data Download from 28+ public databases, scrape 10 competitor platforms, or pull per-record APIs.

bioingest-cli download-all --max-size 2gb
bioingest-cli scrape competitor quanterix
bioingest-cli pull pubmed --query "EGFR biomarker" --limit 50

2

### Transform & Clean Run R/Python scripts to merge, normalize, and prepare data for publishing.

bioingest-cli create-views       # merge mapping tables
Rscript scripts/process_hpa_data.r

The TUI (② Transform) lets you browse, preview, and run scripts interactively.

3

### Publish to S3 Convert to Parquet, upload, create Athena tables — one command.

bioingest-cli publish --crawl              # all sources
bioingest-cli publish --source uniprot     # just one
bioingest-cli publish --dry-run            # preview only

4

### Query & Explore SQL any table. Compare competitor panels in the [Overlap Matrix](../data-explorer/overlap.md).

SELECT analyte, llod_pg_ml
FROM bioingest.competitors_msd__msd_assays
WHERE analyte LIKE '%IL-%';

---

KG Ingestion (⑤)

Build a knowledge graph from PDFs, PubMed, bioRxiv, or local files:

⑤ KG Ingestion — Build Knowledge Graph
────────────────────────────────────────
  Ingest Local PDFs           multi-strategy PDF extraction + LLM
  Ingest PubMed Abstracts     fetch + extract from PubMed
  Ingest bioRxiv Preprints    fetch + extract from bioRxiv API
  Ingest PMC Full-Text        fetch + extract from PMC
  Ingest Local TSV/CSV        tabular data → KG triples
  Load Structured Data        STRING + Reactome + DISEASES → Neptune
  Score Evidence              cross-source validation on edges
  Bulk Ingest (20 queries)    2000 papers across 20 queries
  View Last Run Report        show results from last ingestion run

The pipeline runs: fetch → extract → chunk → KG (LLM) → resolve → write → embed

Each source type is optimized: PubMed uses abstracts via Entrez API, bioRxiv fetches preprints, PMC gets full-text XML, and local PDFs use multi-strategy extraction (pymupdf → pdftotext → vision LLM fallback).

---

Publication Timeline (⑥)

Track how long papers take from preprint (bioRxiv) to peer-reviewed publication:

⑥ Publication Timeline KG

  Enter search terms (comma-separated), or 'gary' for eval entity list:
  Queries: EGFR, IL-6, TP53
  Max preprints per query [20]: 20

  Running 3 queries, max 20 preprints each...

    [1/3] fetch: EGFR
    [2/3] fetch: IL-6
    [3/3] fetch: TP53

  Results:
    Preprints:    60
    Publications: 24 (40%)
    Authors:      483
    Journals:     18
    Median days:  248

    Saved: data/extractions/pub_timeline/pub_timeline_20260630.json

See Publication Timeline KG for schema, derived metrics, and roadmap.

---

Pipeline Status (⑦)

Overview of local data, AWS infrastructure state, and last ingestion run:

⑦ Pipeline Status Overview

  Local Data (data/bulk/)
    Sources: 34  |  Files: 412
    ✓ uniprot               15 files
    ✓ competitors_msd        3 files
    ✓ competitors_nomic      4 files
    ✓ reactome               2 files
    ✓ string                 3 files

  Mapping Tables
    ✓ protein_map     26,499 rows
    ✓ disease_map     31,884 rows
    ✓ drug_map        42,939 rows

  Last Ingestion Run
    Run:   2026-06-28_143022
    Docs:  49  Nodes: 461  Rels: 312

  Graph Triples (mapping_graph.jsonl)
    ✓ 201,322 triples

---

Eval — Extraction Benchmark (⑧)

The TUI includes a built-in evaluation mode that launches graphrag_eval to run the extraction pipeline on real papers and compare against a curated gold standard.

Select ⑧ Eval from the main menu:

⑧ Eval (graphrag_eval)

  Options:
    1. Live Demo — extract from a single paper, compare to gold standard
    2. Run Benchmark — multi-paper evaluation (modes, DPI, models)
    3. Full TUI — launch graphrag_eval interactive TUI

  Option [1/2/3]: 1

Live Demo

  Live Demo — Single Paper Extraction

  Pick a paper:
    * 1 [28 proteins] Predicting antipsychotic responsiveness...
  * = recommended (100% recall in benchmark)

  Mode [1/2] (1): 1  (OCR+LLM)
  Model: llama3-3-70b
  Guided walkthrough? [y/n]: y

Step-by-step pipeline output:

  Step 1/5: Loading paper from cache
  PDF: a69ce6368f.pdf | Gold: 28 proteins, 1 diseases

  Step 2/5: Extracting text from PDF
  Using pypdfium2 to extract text from all pages
    -> Extracted 43,848 characters

  Step 3/5: Chunking + LLM extraction with gleaning
  Splitting into 512-token chunks, sending to llama3-3-70b via Bedrock
    -> Found 51 proteins, 17 diseases

  Step 4/5: Comparing to gold standard
  Normalizing names (IL-6=il6), expanding aliases (CXCL8=IL-8)

  Step 5/5: Results
  Completed in 33.9s | Tokens: in=19,124 out=3,542

Results

  ============================================================
  Results (33.9s) — llama3-3-70b
  ============================================================
                    Summary
  +-------------------+--------------------------+
  | Protein Recall    |        100% (34/34)       |
  | Protein Precision |                      59% |
  | Protein F1        |                      74% |
  | Disease Recall    |                 0% (0/1) |
  | Extracted         | 51 proteins, 17 diseases |
  | Tokens            |      in=19,124 out=3,542 |
  +-------------------+--------------------------+

          Proteins: Gold vs Extracted
  +--------------+------+-----------+--------+
  | Entity       | Gold | Extracted | Status |
  +--------------+------+-----------+--------+
  | CCL19        | yes  |    yes    | MATCH  |
  | CCL3         | yes  |    yes    | MATCH  |
  | CD5          | yes  |    yes    | MATCH  |
  | CDCP1        | yes  |    yes    | MATCH  |
  | CSF-1        | yes  |    yes    | MATCH  |
  | CST5         | yes  |    yes    | MATCH  |
  | DNER         | yes  |    yes    | MATCH  |
  | FGF-21       | yes  |    yes    | MATCH  |
  | FGF-23       | yes  |    yes    | MATCH  |
  | Flt3L        | yes  |    yes    | MATCH  |
  | HGF          | yes  |    yes    | MATCH  |
  | IL-10RB      | yes  |    yes    | MATCH  |
  | IL-12B       | yes  |    yes    | MATCH  |
  | IL-17C       | yes  |    yes    | MATCH  |
  | IL10         | yes  |    yes    | MATCH  |
  | IL18         | yes  |    yes    | MATCH  |
  | IL6          | yes  |    yes    | MATCH  |
  | MCP-3        | yes  |    yes    | MATCH  |
  | MMP-1        | yes  |    yes    | MATCH  |
  | MMP-10       | yes  |    yes    | MATCH  |
  | SCF          | yes  |    yes    | MATCH  |
  | SLAMF1       | yes  |    yes    | MATCH  |
  | TGF-alpha    | yes  |    yes    | MATCH  |
  | TNF          | yes  |    yes    | MATCH  |
  | TNFRSF9      | yes  |    yes    | MATCH  |
  | TNFSF14      | yes  |    yes    | MATCH  |
  | TWEAK        | yes  |    yes    | MATCH  |
  | VEGFA        | yes  |    yes    | MATCH  |
  +--------------+------+-----------+--------+
  | CCL23        |      |    yes    | NEW    |
  | CCL25        |      |    yes    | NEW    |
  | EN-RAGE      |      |    yes    | NEW    |
  | IL-12        |      |    yes    | NEW    |
  | IL-15        |      |    yes    | NEW    |
  | IL-15RA      |      |    yes    | NEW    |
  | Jak-STAT     |      |    yes    | NEW    |
  | MAPK         |      |    yes    | NEW    |
  | ... +10 more |      |           |        |
  +--------------+------+-----------+--------+

          Diseases: Gold vs Extracted
  +------------------+------+-----------+--------+
  | Entity           | Gold | Extracted | Status |
  +------------------+------+-----------+--------+
  | neurology        | yes  |           | MISSED |
  +------------------+------+-----------+--------+
  | adhd             |      |    yes    | NEW    |
  | bipolar disorder |      |    yes    | NEW    |
  | cancer           |      |    yes    | NEW    |
  | covid-19         |      |    yes    | NEW    |
  | depression       |      |    yes    | NEW    |
  +------------------+------+-----------+--------+

100% protein recall — all 28 expected proteins found plus 20 additional valid extractions. Disease recall is 0% because "neurology" is a research area, not a specific disease.

For full benchmark methodology, see graphrag_eval.

---

Minimal .env

# AWS (for publish + query)
AWS_PROFILE=dsinternal
AWS_REGION=eu-north-1

# PubMed (optional, increases rate limit)
ENTREZ_API_KEY=

# Firecrawl (optional, for scrape commands)
FIRECRAWL_API_KEY=fc-your-key

# KG construction (optional)
NEPTUNE_ENDPOINT=cluster.xxx.neptune.amazonaws.com

---

Next Steps

• Data Explorer — Browse all datasets, preview, download
• Platform Overlap — Compare proteins across 10 assay platforms
• Publication Timeline — Preprint-to-publication lifecycle tracking
• Product Spec — Full system architecture and component details