Skip to content

Development

Day 1 Setup

# Clone
git clone https://github.com/dcolinmorgan/graphrag_api.git
cd graphrag-api

# Install uv (if not installed)
pip install uv

# Install all dependencies
uv sync

# Copy environment config
cp .env.example .env
# Edit .env: set NEO4J_PASSWORD, API_AUTH_TOKEN at minimum

# Start Neo4j + Redis
docker compose --profile db up -d
docker compose --profile api run --rm redis

# Verify
uv run pytest -x -q  # Should pass ~2000+ unit tests in <3 min

Contributing

Code Style

  • Ruff for linting and formatting (88-char lines, double quotes, spaces)
  • mypy strict mode (disallow_untyped_defs, disallow_incomplete_defs)
  • Import paths: use src., api., pipeline. as top-level packages
  • Never use stale prefixes like src.api., src.ingest_pipeline.

Key Conventions

Convention Details
Factory pattern src/factories/ creates database, LLM, embedding, agent instances
Domain routers Each API domain has its own router with a routes/ subdirectory
Shared API code Cross-cutting concerns in api/shared/
CLI entry points ingest_main.py, enrichment_main.py, parallel_ingest.py at top level
CDK is isolated cdk_resources/ has its own pyproject.toml and uv.lock
Always use uv run Never bare python or pip install
Test everything New code needs tests — unit at minimum, property-based if possible

Running Checks

uv run ruff check .              # Lint
uv run ruff format .             # Format
uv run mypy src/ api/ pipeline/  # Type check (strict)
uv run pytest                    # Unit tests (parallel, ~3 min)
uv run pytest -m properties      # Property-based tests
uv run pytest -m integration     # Integration tests (requires Neo4j)

PR Workflow

  1. Branch from main: git checkout -b feature/your-feature
  2. Write code + tests
  3. Run uv run ruff check . && uv run ruff format . && uv run pytest -x
  4. Push and open PR against main
  5. CI runs lint + 4 parallel test categories (unit, integration, property-based, slow)
  6. All checks green → merge

CI Pipeline

GitHub Actions (.github/workflows/ci.yml):

Job What Timeout
lint ruff check + ruff format --check 3 min
test (unit) -m "not integration and not slow and not properties" 5 min
test (integration) -m integration 10 min
test (property-based) -m properties 10 min
test (slow) -m slow 15 min

Triggered on: push to main, PRs to main, manual dispatch.

Other workflows: cd.yml (deploy), docs.yml (wiki), eval.yml (accuracy benchmarks), e2e.yml (Playwright).

Onboarding New Developers

Nine self-contained work packages ranked by independence and impact. Start with any rated 9+/10 independence:

# Project Independence Good First?
1 Evaluation Module Hardening 10/10 ✅ Learn all agents
2 MCP Server Expansion 10/10 ✅ Simple pattern, fast wins
3 Fine-Tuned SLMs 9/10 ML background needed
4 3D Protein Structure 9/10 Bio background, greenfield
5 DisGeNET/OpenTargets Integration 9/10 ✅ Clear API pattern
6 Evidence Weighting 8/10 Domain understanding needed
7 OpenAPI Type Generation 8/10 ✅ Mechanical refactoring
8 Playwright E2E 9/10 Frontend-leaning
9 PMC Full-Text Pipeline 7/10 Touches core pipeline

Full details: Modular Projects for New Hires

  1. Day 1: Clone, setup, run tests. Read this wiki cover-to-cover.
  2. Day 2: Build a small KG (follow Getting Started). Query it via curl.
  3. Day 3: Read Architecture and trace a query through the code.
  4. Day 4: Pick a project from the table above. Read its linked doc.
  5. Day 5: Ship your first PR (even if small).

Project Structure Overview

.
├── api/                  # FastAPI REST API (domain-based routers)
│   ├── ingestion/        #   /v1/ingestion/*
│   ├── query/            #   /v1/sessions/*, /v1/evidence, /v1/communities
│   ├── workspace/        #   /v1/cells/*, /v1/my-files/*, /v1/auto-discovery
│   ├── platform/         #   /health, /v1/metrics/*, /v1/dashboard/*
│   └── shared/           #   middleware, security, rate limiting, models
├── src/                  # Core business logic
│   ├── agents/           #   TwoPhase, Dynamic, Neo4j, LangGraph agents
│   ├── services/         #   By domain: query, ingestion, workspace, platform, sdcg
│   ├── factories/        #   DI factories: database, LLM, embedding, agent
│   ├── kg_tools/         #   Tool registry + CLI + MCP + LangChain adapter
│   ├── mcp_server/       #   FastMCP server for AI assistants
│   ├── evaluation/       #   Eval framework: scorers, runners, result store
│   └── ...               #   cache, security, ml, models, utils
├── pipeline/             # Ingestion & processing
│   ├── ingest/           #   Fetchers, chunkers, extractors, KG pipeline
│   ├── enrichment/       #   CSV/Parquet column analysis + annotation
│   └── processors/       #   34 modules: resolution, enrichment, PPI, PDF
├── tests/                # 2652 tests
├── scripts/              # Operational scripts (incremental update, validation)
├── evaluation/           # Eval harness + autoresearch optimizer
├── cdk_resources/        # AWS CDK (separate project)
├── ingest_main.py        # CLI: literature ingestion
├── enrichment_main.py    # CLI: tabular enrichment
└── parallel_ingest.py    # CLI: parallel multi-source ingestion

Environment Variables

Key variables in .env (full list in .env.example):

Variable Required Description
NEO4J_URI Yes Bolt connection string
NEO4J_USER Yes Neo4j username
NEO4J_PASSWORD Yes Neo4j password
API_AUTH_TOKEN Yes Bearer token for API auth
REDIS_HOST No Redis host (default: localhost)
DISABLE_REDIS No Set true for in-memory fallback
BEDROCK_MODEL_ID No AWS Bedrock model for production LLM
BEDROCK_REGION No AWS region for Bedrock
ENABLE_MULTIMODAL_PROCESSING No Enable vision model for PDFs
ENABLE_PII_DETECTION No Enable PII scanning on queries
OTEL_ENABLED No Enable OpenTelemetry tracing

📌 Note (June 2026): Project 5 (DisGeNET/OpenTargets) scope has been reduced — ChEMBL drug-target data is now available via Science Skills integration. STRING PPI data is also now queryable for Project 2 (MCP Server). See the full document for details.

Project Roadmap

6-Month Roadmap (April–September 2026)

Phase Weeks Focus
Stabilize 1–8 Router extraction, CI, onboarding, frontend cleanup
Accelerate 9–18 Async pipeline, entity resolution, data sources, observability
Scale 19–26 Auth, RBAC, Neptune, export, backup

See 6-Month Roadmap for sprint-by-sprint breakdown.

Restructuring Plan

The codebase is being reorganized into clearer domain boundaries:

Domain Scope License
Q&A side agents, KG tools, query service, memory, confidence scoring Proprietary
Ingestion side pipeline, processors, enrichment, fetchers Open-source candidate
Shared core, models, cache, factories, security, utils Shared

See Restructuring Sketch.

Entity Extraction Strategy

Hybrid "extract broadly, filter precisely":

  1. LLM: broad entity extraction from text (permissive by design)
  2. Ontology filter: validates against UniProt/MONDO, removes spurious entities
  3. User control: configurable entity types, filtering strictness, custom rules

See User vs LLM Decisions.

Status Reports & References