Gold Standard: Gary's curated PUBLICATIONS_SUMMARIES CSV. Each paper has manually annotated disease areas and protein/marker names used as ground truth labels.
Pipeline:
- Paper Selection — Papers sorted by label richness (most proteins+diseases first). Paywalled sources excluded. Pool of 3x target loaded, filtered to N papers with successful full-text fetch.
- OCR/Parsing — Each method (markitdown, pymupdf, docling) converts the same cached raw file (PDF or HTML) to text. All methods see identical source content for fair comparison.
- LLM Extraction — Each model extracts proteins and diseases from parsed text. Run 10x per paper to measure reproducibility.
- Scoring — Extracted entities compared against gold labels. Protein matching uses normalization (strip hyphens, alias dictionary). Disease matching maps specific diseases to gold categories.
Metrics:
- Precision = |extracted ∩ gold| / |extracted| — "of what we found, how much was correct?"
- Recall = |extracted ∩ gold| / |gold| — "of what Gary listed, how much did we find?"
- F1 = harmonic mean of precision and recall
- Reproducibility = avg pairwise Jaccard similarity across 10 runs — "how consistent is the model?"
- Cross-Model Agreement = Jaccard between different models on same paper
Note: We typically extract MORE entities than Gary listed (he noted highlights, not exhaustive lists). High precision + moderate recall is expected.