Skip to content

Entity Resolution

How bioingest resolves and deduplicates protein/disease/drug entities across different naming conventions.

The Problem

The same protein appears with different names across sources:

Source Name for Interleukin-6
PubMed paper "IL-6"
UniProt "Interleukin-6"
MSD catalog "IL-6"
Olink panel "IL6"
LLM extraction "interleukin 6"

Without resolution, these create 5 separate nodes in the graph instead of 1.

Resolution Strategies (applied in order)

Strategy 1: Exact Normalized Match

Case-insensitive, whitespace-stripped comparison.

"IL-6" and "il-6" → same entity.

Strategy 2: Informal Name Map

~16 colloquial names used on competitor panels that don't exist in any gene/protein database. These are hand-maintained for chemokine common names where vendors use non-standard nomenclature:

"rantes"    → "ccl5"       # Common name, not in HGNC
"eotaxin"   → "ccl11"      # MSD/Quanterix naming
"mcp-1"     → "ccl2"       # MSD catalog format
"mip-1a"    → "ccl3"       # Competitor panel name
"ip-10"     → "cxcl10"     # Common abbreviation
"gro-alpha" → "cxcl1"      # MSD format
"fgf-basic" → "fgf2"       # MSD/Bio-Techne convention

Only entries where the colloquial name cannot be found in HGNC or UniProt belong here.

Strategy 3: Abbreviation Expansion

If a short all-caps name matches the initials of a longer name already in the graph:

"EGFR" → initials of "epidermal growth factor receptor" → merged.

Strategy 4: HGNC + UniProt Extended Synonyms (~130K pairs)

Loaded lazily on first use from two sources:

HGNC Complete Set (~80K alias→symbol mappings):

  • Downloaded automatically from ftp.ebi.ac.uk/pub/databases/genenames/hgnc/tsv/hgnc_complete_set.txt on first use
  • Parses prev_symbol and alias_symbol columns → maps to approved symbol
  • Covers all retired gene names, aliases, and historical symbols
prev_symbol "ERBB1" → symbol "EGFR"
alias_symbol "HER1" → symbol "EGFR"
prev_symbol "TNFA" → symbol "TNF"
alias_symbol "TNFSF2" → symbol "TNF"

UniProt SwissProt (~50K gene + protein name mappings):

  • Source: data/bulk/uniprot/swissprot_tsv.tsv (from bioingest download uniprot)
  • Gene Names column: all synonyms mapped to primary gene symbol
  • Protein names column: short protein name mapped to gene symbol
Gene Names: "EGFR ERBB1 HER1" → ERBB1 → EGFR, HER1 → EGFR
Protein names: "Interleukin-6" → primary gene: IL6

Combined: ~130,000 synonym pairs covering every human gene symbol alias and protein name variant.

Strategy 5: Fuzzy Matching (rapidfuzz)

For names that don't match exactly or via synonyms, use Jaro-Winkler similarity with a high threshold (≥ 92%):

"Interleukin 6" vs "Interleukin-6" → 96% match → merged
"TNF alpha" vs "TNF-alpha" → 94% match → merged
"VEGF-A" vs "VEGFA" → 92% match → merged

Falls back gracefully if rapidfuzz is not installed (strategy skipped).

Configuration

# Disable entity resolution entirely
export ENABLE_ENTITY_RESOLUTION=false

# In code
resolver = EntityResolver(
    enable_fuzzy=True,        # enable/disable step 5
    data_dir=Path("data"),    # path to UniProt data for step 4
)

Where the Mappings Come From

No LLMs are used for entity resolution — it's entirely deterministic and reproducible.

Strategy 2: Informal Name Map (hand-curated)

Maintained in bioingest/pipeline/entity_resolver.py as a Python dict (~16 entries). Only colloquial chemokine/growth factor names that cannot be resolved via any database:

"rantes"    → "ccl5"       # No database maps this common name
"eotaxin"   → "ccl11"      # Vendor-specific naming
"mcp-1"     → "ccl2"       # MSD format
"fgf-basic" → "fgf2"       # Bio-Techne convention

Added only when a competitor panel name cannot be resolved by HGNC or UniProt lookups.

Strategy 3: Abbreviation Expansion (algorithmic)

No data file — pure algorithm. Checks if initials of an existing longer name match the short name.

Strategy 4: HGNC + UniProt (auto-downloaded databases)

HGNC (Human Gene Nomenclature Committee):

  • Source: https://ftp.ebi.ac.uk/pub/databases/genenames/hgnc/tsv/hgnc_complete_set.txt
  • Auto-downloaded on first use to data/bulk/hgnc/hgnc_complete_set.tsv (~5MB)
  • Authority for approved human gene symbols
  • Provides: prev_symbol → symbol and alias_symbol → symbol mappings
  • Coverage: ~80,000 alias pairs for ~43,000 genes

UniProt SwissProt:

  • Source: data/bulk/uniprot/swissprot_tsv.tsv (downloaded by bioingest download uniprot)
  • Gene Names: all alternative gene symbols → primary (~50K pairs)
  • Protein names: short recommended name → primary gene symbol
  • Coverage: all 20,000+ human reviewed protein entries

Strategy 5: Fuzzy Matching (rapidfuzz library)

Source: no data file — uses the rapidfuzz Python package (C-optimized string similarity).

Compares new entity names against names already in the graph using Jaro-Winkler weighted ratio. Only merges at ≥92% similarity:

"Interleukin 6"  vs "Interleukin-6"    → 96% → merge
"TNF alpha"      vs "TNF-alpha"        → 94% → merge  
"BRCA 1"         vs "BRCA1"            → 93% → merge
"Random Name"    vs "Other Name"       → 45% → NO merge

This catches formatting variants (hyphens, spaces, special characters) that the other strategies miss.

What is NOT used

  • No LLMs for entity resolution (fully deterministic)
  • No external APIs called during resolution
  • No machine learning models
  • No manual annotation per paper

The entity resolver works within a single ingestion run (merging LLM-extracted entities). The mapping tables (protein_map, disease_map) work across sources at the database level.

LLM extracts "IL-6", "Interleukin 6", "il6" from a paper
    ↓ Entity Resolver (steps 1-5)
One node: "Interleukin 6"
    ↓ Enrichment (protein_map lookup)
Annotated: uniprot_id=P05231, ensembl=ENSG00000136244
    ↓ Written to Neptune
(:Protein {id: "P05231", name: "Interleukin 6", gene: "IL6"})

Impact

Without resolution With resolution
461 entities from 49 papers ~350 unique entities (24% reduction)
Duplicate nodes for same protein Single canonical node per protein
No UniProt IDs UniProt/MONDO IDs via enrichment
Can't join to structured data Joins to STRING, Reactome, DISEASES

Relationship to graphrag_api Consolidation

Entity resolution in bioingest handles pre-write normalization — merging names before they reach the graph. The graphrag_api entity consolidation handles post-write deduplication — merging nodes already in the database.

Phase Tool When Strategy
Pre-write bioingest EntityResolver During LLM extraction Synonym maps, abbreviation, fuzzy (in-memory)
Incremental graphrag_api IncrementalConsolidator Every 25 chunks Name-based (Cypher MERGE)
Final graphrag_api EntityResolver After ingestion UniProt ID, fuzzy, MONDO ontology

The two systems are complementary:

  1. bioingest reduces duplicates by ~24% before writing (cheap, in-memory)
  2. graphrag_api catches anything that slipped through and adds ontology hierarchies (requires DB queries)

Both share the same underlying data: UniProt synonyms and MONDO disease ontology. bioingest loads them from local TSV files; graphrag_api queries them from the graph or loads from OBO files.