KG Pipeline Consistency Test Results

Test Date: 2025-08-06 12:33:44 Configuration: 5 abstracts × 3 runs = 15 total extractions

Overall Summary

Metric	Value
Success Rate	100.0%
Entity Consistency	14.9%
Relationship Consistency	9.5%
Overall Consistency Score	12.2%
Mean Processing Time	31.70s ± 14.89s
Total Processing Time	475.56s

Per-Abstract Performance

PMID	Title	Success Rate	Entities (Avg)	Entity Consistency	Relationships (Avg)	Relationship Consistency	Processing Time
36810097	The Alzheimer's disease-linked protease BACE1 modu...	100.0%	11.3	33.3%	5.7	30.0%	17.21s ± 2.69s
32630818	IL-6 Trans-Signaling in the Brain Influences the M...	100.0%	10.0	5.6%	8.0	11.8%	22.83s ± 0.54s
39206334	Genetically proxied IL-6 signaling and risk of Alz...	100.0%	10.0	13.6%	6.7	0.0%	27.31s ± 4.55s
37697347	H3K18 lactylation of senescent microglia potentiat...	100.0%	17.0	21.9%	9.3	4.3%	34.84s ± 9.86s
37600508	Exercise therapy to prevent and treat Alzheimer's ...	100.0%	28.7	0.0%	30.7	1.3%	56.33s ± 6.32s

Detailed Analysis by Abstract

PMID 36810097

Title: The Alzheimer's disease-linked protease BACE1 modulates neuronal IL-6 signaling through shedding of the receptor gp130

Statistics

Metric	Value
Success Rate	100.0% (3/3 runs)
Entity Count (Mean ± SD)	11.3 ± 3.1
Entity Range	8 - 14
Entity Consistency	33.3% (6/18)
Relationship Count (Mean ± SD)	5.7 ± 1.2
Relationship Range	5 - 7
Relationship Consistency	30.0% (3/10)
Processing Time (Mean ± SD)	17.21s ± 2.69s
Coefficient of Variation	0.156

Most Frequent Entities

Entity	Frequency	Consistency
BACE1	3/3	✅ 100%
gp130	3/3	✅ 100%
IL-6	3/3	✅ 100%
Alzheimer's disease	3/3	✅ 100%
BACE inhibitors	3/3	✅ 100%
SEZ6	3/3	✅ 100%
neuroinflammation	2/3	⚠️ 66.7%
signaling	2/3	⚠️ 66.7%
gp130/IL6ST	2/3	⚠️ 66.7%
growth-factor withdrawal	2/3	⚠️ 66.7%

Most Frequent Relationships

Relationship	Frequency	Consistency
BACE1-MODULATES->Alzheimer's disease	3/3	✅ 100%
BACE1-CLEAVES->gp130/IL6ST	3/3	✅ 100%
exercise-PREVENTS->cognitive decline	3/3	✅ 100%
BACE inhibitors-TARGET->BACE1	2/3	⚠️ 66.7%
BACE1 inhibition-ASSOCIATED_WITH->cognitive worsening	1/3	⚠️ 33.3%
BACE1 inhibitors-TARGET->BACE1	1/3	⚠️ 33.3%
SEZ6-REDUCED BY->BACE1 inhibitors	1/3	⚠️ 33.3%
gp130/IL6ST-SUBSTRATE OF->BACE1	1/3	⚠️ 33.3%
gp130-CLEAVES BY->BACE1	1/3	⚠️ 33.3%
BACE1 inhibition-ASSOCIATED_WITH->non-progressive cognitive worsening	1/3	⚠️ 33.3%

Run-by-Run Results

Run	Success	Entities	Relationships	Processing Time	Notes
1	✅	8	5	14.31s	✅ Success
2	✅	14	7	19.61s	✅ Success
3	✅	12	5	17.70s	✅ Success

---

PMID 32630818

Title: IL-6 Trans-Signaling in the Brain Influences the Metabolic Phenotype of the 3xTg-AD Mouse Model of Alzheimer's Disease

Statistics

Metric	Value
Success Rate	100.0% (3/3 runs)
Entity Count (Mean ± SD)	10.0 ± 0.0
Entity Range	10 - 10
Entity Consistency	5.6% (1/18)
Relationship Count (Mean ± SD)	8.0 ± 1.7
Relationship Range	7 - 10
Relationship Consistency	11.8% (2/17)
Processing Time (Mean ± SD)	22.83s ± 0.54s
Coefficient of Variation	0.024

Most Frequent Entities

Entity	Frequency	Consistency
Alzheimer	3/3	✅ 100%
IL-6	2/3	⚠️ 66.7%
gp130	2/3	⚠️ 66.7%
Alzheimer's disease	2/3	⚠️ 66.7%
obesity	2/3	⚠️ 66.7%
signaling	2/3	⚠️ 66.7%
neuroinflammation	2/3	⚠️ 66.7%
receptors	2/3	⚠️ 66.7%
enzymes	2/3	⚠️ 66.7%
Note	2/3	⚠️ 66.7%

Most Frequent Relationships

Relationship	Frequency	Consistency
BACE1-MODULATES->Alzheimer's disease	3/3	✅ 100%
sgp130-INHIBITS->IL-6 trans-signaling	3/3	✅ 100%
exercise-PREVENTS->cognitive decline	2/3	⚠️ 66.7%
IL-6-MODULATES->obesity	2/3	⚠️ 66.7%
IL-6-MODULATES->insulin resistance	2/3	⚠️ 66.7%
IL-6-MODULATES->Alzheimer's disease	1/3	⚠️ 33.3%
IL-6-CLEAVES->gp130	1/3	⚠️ 33.3%
IL-6 trans-signaling-MODULATES->obesity progression	1/3	⚠️ 33.3%
sgp130Fc mice-TARGETS->IL-6 trans-signaling	1/3	⚠️ 33.3%
IL-6 trans-signaling-TARGETS->gp130	1/3	⚠️ 33.3%

Run-by-Run Results

Run	Success	Entities	Relationships	Processing Time	Notes
1	✅	10	7	22.92s	✅ Success
2	✅	10	7	22.25s	✅ Success
3	✅	10	10	23.33s	✅ Success

---

PMID 39206334

Title: Genetically proxied IL-6 signaling and risk of Alzheimer's disease and lobar intracerebral hemorrhage: A drug target Mendelian randomization study

Statistics

Metric	Value
Success Rate	100.0% (3/3 runs)
Entity Count (Mean ± SD)	10.0 ± 0.0
Entity Range	10 - 10
Entity Consistency	13.6% (3/22)
Relationship Count (Mean ± SD)	6.7 ± 4.0
Relationship Range	2 - 9
Relationship Consistency	0.0% (0/20)
Processing Time (Mean ± SD)	27.31s ± 4.55s
Coefficient of Variation	0.167

Most Frequent Entities

Entity	Frequency	Consistency
Here	3/3	✅ 100%
Alzheimer	3/3	✅ 100%
Note	3/3	✅ 100%
Disease	2/3	⚠️ 66.7%
And	2/3	⚠️ 66.7%
Protein	1/3	⚠️ 33.3%
Molecular	1/3	⚠️ 33.3%
Genetic	1/3	⚠️ 33.3%
Drugs	1/3	⚠️ 33.3%
Corrected	1/3	⚠️ 33.3%

Most Frequent Relationships

Relationship	Frequency	Consistency
The source entity-MODULATES->or regulates the target entity	1/3	⚠️ 33.3%
The source entity modifies or-AFFECTS->the target entity	1/3	⚠️ 33.3%
IL-6 signaling-MODULATES->CRP	1/3	⚠️ 33.3%
IL-6 receptor gene-UPSTREAM_OF->CRP	1/3	⚠️ 33.3%
CRP-lowering variants near IL-6 receptor gene-TARGET->IL-6 signaling	1/3	⚠️ 33.3%
Genetically upregulated IL-6 receptor-mediated signaling-MODULATES->Alzheimer's disease	1/3	⚠️ 33.3%
Genetically upregulated IL-6 receptor-mediated signaling-MODULATES->lobar intracerebral hemorrhage	1/3	⚠️ 33.3%
CRP-lowering through variation in the gene encoding IL-6 receptor-TARGETS->BACE1 (implied, not directly stated)	1/3	⚠️ 33.3%
IL-6 signaling-MODULATES->amyloid-related pathologies	1/3	⚠️ 33.3%
Genetically upregulated IL-6 receptor-mediated signaling-ASSOCIATED_WITH->lower risk of AD and lobar ICH	1/3	⚠️ 33.3%

Run-by-Run Results

Run	Success	Entities	Relationships	Processing Time	Notes
1	✅	10	2	28.27s	✅ Success
2	✅	10	9	22.36s	✅ Success
3	✅	10	9	31.30s	✅ Success

---

PMID 37697347

Title: H3K18 lactylation of senescent microglia potentiates brain aging and Alzheimer's disease through the NFκB signaling pathway

Statistics

Metric	Value
Success Rate	100.0% (3/3 runs)
Entity Count (Mean ± SD)	17.0 ± 1.7
Entity Range	15 - 18
Entity Consistency	21.9% (7/32)
Relationship Count (Mean ± SD)	9.3 ± 1.5
Relationship Range	8 - 11
Relationship Consistency	4.3% (1/23)
Processing Time (Mean ± SD)	34.84s ± 9.86s
Coefficient of Variation	0.283

Most Frequent Entities

Entity	Frequency	Consistency
Alzheimer's disease	3/3	✅ 100%
BACE1	3/3	✅ 100%
gp130	3/3	✅ 100%
IL-6	3/3	✅ 100%
IL-8	3/3	✅ 100%
Rela (p65)	3/3	✅ 100%
NFκB1(p50)	3/3	✅ 100%
NFκB signaling pathway	2/3	⚠️ 66.7%
APP/PS1	2/3	⚠️ 66.7%
FAD4T	2/3	⚠️ 66.7%

Most Frequent Relationships

Relationship	Frequency	Consistency
exercise-PREVENTS->cognitive decline	3/3	✅ 100%
senescent microglia-MODULATES->brain aging	2/3	⚠️ 66.7%
H3K18 lactylation-POTENTIATES->Alzheimer's disease	2/3	⚠️ 66.7%
H3K18 lactylation-STIMULATES->NFκB signaling pathway	2/3	⚠️ 66.7%
H3K18 lactylation of senescent microglia-POTENTIATES->Alzheimer's disease	1/3	⚠️ 33.3%
NFκB signaling pathway-AFFECTS->senescence-associated secretory phenotype (SASP) components IL-6 and IL-8	1/3	⚠️ 33.3%
H3K18 lactylation of senescent microglia-STIMULATES->NFκB signaling pathway	1/3	⚠️ 33.3%
H3K18 lactylation-BINDS TO->Rela (p65) and NFκB1(p50)	1/3	⚠️ 33.3%
IL-6 and IL-8-UPREGULATES->senescence-associated secretory phenotype (SASP)	1/3	⚠️ 33.3%
NFκB signaling pathway-MODULATES->H3K18 lactylation of senescent microglia	1/3	⚠️ 33.3%

Run-by-Run Results

Run	Success	Entities	Relationships	Processing Time	Notes
1	✅	18	8	29.90s	✅ Success
2	✅	18	11	28.43s	✅ Success
3	✅	15	9	46.19s	✅ Success

---

PMID 37600508

Title: Exercise therapy to prevent and treat Alzheimer's disease

Statistics

Metric	Value
Success Rate	100.0% (3/3 runs)
Entity Count (Mean ± SD)	28.7 ± 16.2
Entity Range	10 - 38
Entity Consistency	0.0% (0/60)
Relationship Count (Mean ± SD)	30.7 ± 3.8
Relationship Range	28 - 35
Relationship Consistency	1.3% (1/75)
Processing Time (Mean ± SD)	56.33s ± 6.32s
Coefficient of Variation	0.112

Most Frequent Entities

Entity	Frequency	Consistency
Protein	5/3	⚠️ 166.7%
BACE1	2/3	⚠️ 66.7%
gp130	2/3	⚠️ 66.7%
IL-6	2/3	⚠️ 66.7%
BACE inhibitors	2/3	⚠️ 66.7%
NGF	2/3	⚠️ 66.7%
p-AKT	2/3	⚠️ 66.7%
p-TrkB	2/3	⚠️ 66.7%
p-PKC	2/3	⚠️ 66.7%
BDNF	2/3	⚠️ 66.7%

Most Frequent Relationships

Relationship	Frequency	Consistency
exercise training-DECREASES->mitochondrial fragmentation	3/3	✅ 100%
BACE1-MODULATES->Alzheimer's disease	2/3	⚠️ 66.7%
BACE inhibitors-TARGET->BACE1	2/3	⚠️ 66.7%
exercise therapy-DECREASES->Aβ	2/3	⚠️ 66.7%
exercise therapy-DECREASES->tau phosphorylation	2/3	⚠️ 66.7%
aerobic exercise-PREVENTS->hippocampal volume reduction	2/3	⚠️ 66.7%
aerobic exercise-PREVENTS->spatial memory reduction	2/3	⚠️ 66.7%
aerobic exercise-INCREASES->synaptic flexibility	2/3	⚠️ 66.7%
exercise training-DECREASES->DRP1	2/3	⚠️ 66.7%
exercise training-DECREASES->FIS1	2/3	⚠️ 66.7%

Run-by-Run Results

Run	Success	Entities	Relationships	Processing Time	Notes
1	✅	38	28	56.66s	✅ Success
2	✅	10	29	49.85s	✅ Success
3	✅	38	35	62.48s	✅ Success

---

Recommendations

❌ Low Consistency - Improvements Needed

• LLM responses are highly variable
• Action Items:
• Review and refine extraction prompts
• Consider ensemble methods or majority voting
• Investigate prompt engineering techniques
• Consider using different LLM model or settings
• Implement consistency checks and validation

Technical Notes

• LLM Model: Local Ollama (llama3.1:8b-instruct-q8_0)
• Extraction Method: Direct LLM prompting with JSON response parsing
• Fallback Handling: Regex-based extraction for malformed JSON responses
• Consistency Measurement: Exact string matching for entities/relationships across runs

---

Report generated on 2025-08-06 12:51:54