BioIngest¶
A self-service data platform for collecting, cleaning, and exploring biomedical datasets.
BioIngest automates the tedious work of downloading data from 30+ public databases, scraping competitor catalogs, running cleaning scripts, and publishing everything to a shared data lake. Product managers and researchers can browse, compare, and query data without writing code.
Quick links¶
Browse S3 Bucket :material-folder-open:{ .md-button } Browse GitHub :material-github:{ .md-button } Open Athena :material-database-search:{ .md-button } Request a Source :material-plus-circle:{ .md-button .md-button--primary }
Data Explorer¶
Browse 28 biomedical data sources, 66 queryable tables, and 3 cross-reference mapping tables.
Click :material-table-eye: to preview real columns and rows before downloading.
Browse Data¶
Proteins & Targets
| Dataset | Description | :material-table-eye: | GitHub | S3 |
|---|---|---|---|---|
uniprot/swissprot_tsv |
570K reviewed proteins (15 cols) | :material-table-eye: | :material-github: | :material-cloud: |
uniprot/idmapping_human |
UniProt cross-references (5 cols) | :material-table-eye: | :material-github: | :material-cloud: |
string/protein_links |
Protein-protein interactions (3 cols) | :material-table-eye: | :material-github: | :material-cloud: |
interpro/protein2ipr |
Protein domain families (6 cols) | :material-table-eye: | :material-github: | :material-cloud: |
chembl/uniprot_mapping |
ChEMBL target IDs (4 cols) | :material-table-eye: | :material-github: | :material-cloud: |
complex_portal/homo_sapiens |
Protein complexes (19 cols) | :material-table-eye: | :material-github: | :material-cloud: |
markerdb/proteins |
4K biomarker proteins (12 cols) | :material-table-eye: | :material-github: | :material-cloud: |
Disease & Pathway Associations
| Dataset | Description | :material-table-eye: | GitHub | S3 |
|---|---|---|---|---|
diseases/knowledge |
Disease-gene associations (7 cols) | :material-table-eye: | :material-github: | :material-cloud: |
diseases/experiments |
Experiment-based associations | :material-table-eye: | :material-github: | :material-cloud: |
reactome/UniProt2Reactome |
2.5M pathway memberships (6 cols) | :material-table-eye: | :material-github: | :material-cloud: |
ttd/targets |
Therapeutic targets & drugs | :material-table-eye: | :material-github: | :material-cloud: |
ttd/drugs |
Drug-target-indication triples | :material-table-eye: | :material-github: | :material-cloud: |
clinvar/variant_summary |
Variant-disease significance | :material-table-eye: | :material-github: | :material-cloud: |
gtex/median_tissue_tpm |
Tissue expression matrix | :material-table-eye: | :material-github: | :material-cloud: |
Ontologies
| Dataset | Description | :material-table-eye: | GitHub | S3 |
|---|---|---|---|---|
mondo/mondo |
47K disease terms + hierarchy | :material-table-eye: | :material-github: | :material-cloud: |
disease_ontology/doid |
12K Disease Ontology terms | :material-table-eye: | :material-github: | :material-cloud: |
efo/efo |
50K experimental factors | :material-table-eye: | :material-github: | :material-cloud: |
gene_ontology/go_basic |
45K Gene Ontology terms | :material-table-eye: | :material-github: | :material-cloud: |
mesh/descriptors |
MeSH medical vocabulary | :material-table-eye: | :material-github: | :material-cloud: |
icd/icd10cm |
ICD-10-CM diagnosis codes | :material-table-eye: | :material-github: | :material-cloud: |
Competitors & Assay Platforms
| Dataset | Description | :material-table-eye: | GitHub | S3 |
|---|---|---|---|---|
competitors_msd/msd_assays |
MSD assay specs (LOD, range) | :material-table-eye: | :material-github: | :material-cloud: |
competitors_msd/msd_products_full |
MSD full catalog (2640 products, 661 analytes) | :material-table-eye: | :material-github: | :material-cloud: |
competitors_quanterix/assays |
Quanterix Simoa specs (8 cols) | :material-table-eye: | :material-github: | :material-cloud: |
competitors_alamar/all_targets |
Alamar NULISAseq targets (26 cols) | :material-table-eye: | :material-github: | :material-cloud: |
competitors_nomic/omni_1000 |
Nomic Omni 1000 targets (990) | :material-table-eye: | :material-github: | :material-cloud: |
competitors_nomic/core_immune |
Nomic Core Immune (284) | :material-table-eye: | :material-github: | :material-cloud: |
competitors_nomic/core_cardiomet |
Nomic Core CardioMet (286) | :material-table-eye: | :material-github: | :material-cloud: |
competitors_nomic/flex |
Nomic Flex (1056) | :material-table-eye: | :material-github: | :material-cloud: |
somalogic/somascan_11k |
SomaScan 11K menu | :material-table-eye: | :material-github: | :material-cloud: |
Olink & HPA Proteomics
| Dataset | Description | :material-table-eye: | GitHub | S3 |
|---|---|---|---|---|
olink/explore_ht |
Explore HT full panel (5,416 proteins) | — | :material-github: | :material-cloud: |
olink/explore_3072 |
Explore 3072 (8 panels × ~368) | — | :material-github: | :material-cloud: |
olink/reveal |
Reveal panel (1,034 proteins) | — | :material-github: | :material-cloud: |
olink/flex |
Flex library (197 proteins) | — | :material-github: | :material-cloud: |
olink/target_96 |
11 Target 96 panels (92 proteins each) | — | :material-github: | :material-cloud: |
olink/focus |
Focus panel (33 proteins) | — | :material-github: | :material-cloud: |
platform_overlap |
Reconciled overlap data (24 platforms) | — | :material-github: | :material-cloud: |
hpa_olink/proteinatlas |
HPA full proteomics | :material-table-eye: | :material-github: | :material-cloud: |
Genomics & Variants
| Dataset | Description | :material-table-eye: | GitHub | S3 |
|---|---|---|---|---|
gnomad/constraint |
Gene constraint (pLI, LOEUF) | :material-table-eye: | :material-github: | :material-cloud: |
ensembl/gene_info |
Gene annotations (GRCh38) | :material-table-eye: | :material-github: | :material-cloud: |
dbsnp/variant_freq |
Population variant frequencies | :material-table-eye: | :material-github: | :material-cloud: |
ukb_disease_assoc/pan_ukb |
UK Biobank GWAS | :material-table-eye: | :material-github: | :material-cloud: |
How it works¶
flowchart TB
DB[(Public DBs\nUniProt, STRING, Reactome, ClinVar...)] --> DL
COMP[Competitor Catalogs\nMSD, Quanterix, Nomic, RBM, Alamar, Roche...] --> DL
INT[Internal Files\nOlink panels, assay lists, curated spreadsheets] --> DL
DL[Download / Scrape] --> EDIT[Clean and Transform\nR scripts, merge tables, normalize columns]
EDIT --> PUBLISH[Publish\nconvert to Parquet, upload to S3]
PUBLISH --> GIT[GitHub / Git LFS\nversion-controlled files]
PUBLISH --> S3[(S3 Data Lake\nParquet + Athena SQL)]
GIT --> WIKI[Wiki / Data Explorer\nbrowse, preview, compare]
S3 --> WIKIWhat you can do¶
| Task | How |
|---|---|
| Browse available data | Use the Data Explorer above -- preview columns and rows before downloading |
| Download raw files | bioingest download-all or clone the repo (Git LFS) |
| Compare competitor panels | Platform Overlap Matrix -- see which proteins each platform measures |
| Run cleaning scripts | scripts/ folder -- R/Python scripts for merging and normalizing tables |
| Query with SQL | Open Athena and query any table across all sources |
| Request a new source | Request form -- tell us what you need and we'll add it |
Mapping Tables¶
Cross-reference tables linking all sources through shared identifiers:
| Table | Rows | Cols | What it maps | :material-table-eye: | S3 |
|---|---|---|---|---|---|
| protein_map | 26,499 | 9 | UniProt ↔ Ensembl ↔ STRING ↔ ChEMBL ↔ HGNC | :material-table-eye: | :material-cloud: |
| disease_map | 31,884 | 10 | MONDO ↔ EFO ↔ DOID ↔ MeSH ↔ ICD-10 | :material-table-eye: | :material-cloud: |
| drug_map | 42,939 | 5 | TTD drugs ↔ targets ↔ indications | :material-table-eye: | :material-cloud: |
Don't see what you need?
Request a New Data Source :material-plus-circle:{ .md-button .md-button--primary }
We add new sources regularly. Tell us what database you'd like and we'll build the connector.