Does the study actually say that?
claim-to-source reads a health article, resolves the studies it cites through Europe PMC, and reports what each cited paper does and does not support.
The distinction the tool exists to protect
Three outcomes, and only one of them is a criticism of the article.
- = SUPPORTED the cited paper says this, and none of the named failure modes fired.
- x NOT SUPPORTED the tool read the source and found a specific, named mismatch. Every one of these shows the sentence from the source it rests on.
- ? NOT CHECKED the tool did not read the source: no citation, a citation that resolves to nothing, a fetch that failed, or a record with no abstract. This is a statement about the tool, not about the article. It carries no confidence score and can never carry an accusation.
This run
…
Eight articles written for this repository, each one designed around a single named failure mode, plus one written to be entirely faithful to its sources. The studies they cite are real papers, resolved from a committed Europe PMC cache.
The check worth the most: relative against absolute
“Halves your risk” is a ratio. Whether it matters depends on the baseline, and when the source states the baseline the useful number is exactly computable.
Aggressive blood pressure targets halve your heart risk
| what the article says | 50% lower risk |
|---|---|
| baseline in the source | 2.19% per year (1 in 46) |
| what the article's number implies | 1.095% per year |
| change implied, in points | 1.095 |
| people affected per 10,000 | 110 |
| one event per | 91 people |
| what the study itself found | 2.19% to 1.65% per year, a difference of 0.54 points |
What happened on a real article
Written fixtures only test what their author already thought of, so the tool was pointed at six live NIH news releases. One of them is kept in the repository because of what it found.
A false accusation, at confidence 0.80. The release reports the lifestyle arm of a diabetes-prevention follow-up accurately. The abstract of that paper also contains the sentence “There was no difference between participants in the metformin and placebo groups”, about an arm the release makes no claim about. The contradiction check fired on any null sentence anywhere in an abstract, and so told a reader that an accurate press release contradicted its own source. It now requires the null sentence to share a distinctive term with the claim, and a regression test holds that in place alongside a control proving the check still fires on a claim that really is about the metformin arm.
Eight of ten claims came back NOT CHECKED, and that is the honest answer. A press release puts its reference in a line at the foot of the page, not beside each sentence, and the proximity rule will not reach that far. A tool willing to guess would have attributed ten claims to one paper and judged every one of them.
Every claim, every verdict
causal-from-cohort.html
3 claims · 1 supported · 2 not supported · 0 not checked · 0 omission(s) · exit 1
x NOT SUPPORTED CAUSAL_FROM_OBSERVATIONAL
“Four cups of coffee a day cuts your risk of dying”
the source is observational and reports an association; this sentence states it as cause and effect, with no hedge
Association of coffee drinking with total and cause-specific mortality
x NOT SUPPORTED CAUSAL_FROM_OBSERVATIONAL
“Drinking four or five cups of coffee a day lowers your risk of death by 12%, according to a study of American adults in the New England Journal of Medicine (Freedman et al., 2012).”
the source is observational and reports an association; this sentence states it as cause and effect, with no hedge
Association of coffee drinking with total and cause-specific mortality
= SUPPORTED SUPPORTED_BY_SOURCE
“Coffee may also protect the heart, the researchers noted.”
the cited source states this, and none of the named failure modes fired
Association of coffee drinking with total and cause-specific mortality
contradicted.html
2 claims · 0 supported · 2 not supported · 0 not checked · 4 omission(s) · exit 1
x NOT SUPPORTED DIRECTION_CONTRADICTED
“A shot of beetroot juice before a race improves sprint performance in trained athletes, according to a placebo-controlled trial in the Journal of the International Society of Sports Nutrition (trial).”
the article asserts an effect; the cited study reports that it did not find one
Acute beetroot juice ingestion fails to improve sprint performance and neuromuscular function in trained male sprinters: a randomized, double-blind, placebo-controlled study
x NOT SUPPORTED DIRECTION_CONTRADICTED
“Nitrate in the juice boosts muscle contraction, the authors explain.”
the article asserts an effect; the cited study reports that it did not find one
Acute beetroot juice ingestion fails to improve sprint performance and neuromuscular function in trained male sprinters: a randomized, double-blind, placebo-controlled study
fabricated-attribution.html
1 claims · 0 supported · 1 not supported · 0 not checked · 0 omission(s) · exit 1
x NOT SUPPORTED NOT_IN_SOURCE
“The same trial found that eating more fermented food improves gut microbiome diversity (SPRINT, 2015).”
the cited paper is about something else; nothing in it addresses this sentence
in-vitro-as-human.html
2 claims · 0 supported · 2 not supported · 0 not checked · 0 omission(s) · exit 1
x NOT SUPPORTED IN_VITRO_AS_HUMAN
“Broccoli compound kills breast cancer, study finds”
the cited study treated cells in culture, not people or animals; the article does not say so
Sulforaphane promotes the removal of doxorubicin-induced senescent epithelial and breast cancer cells and activates AhR/CYP1A1-based adaptive response in vitro
x NOT SUPPORTED IN_VITRO_AS_HUMAN
“Sulforaphane, the compound that gives broccoli its bitter edge, kills breast cancer cells, researchers report in Chemico-Biological Interactions (study).”
the cited study treated cells in culture, not people or animals; the article does not say so
Sulforaphane promotes the removal of doxorubicin-induced senescent epithelial and breast cancer cells and activates AhR/CYP1A1-based adaptive response in vitro
mouse-as-human.html
2 claims · 0 supported · 2 not supported · 0 not checked · 0 omission(s) · exit 1
x NOT SUPPORTED SPECIES_MISMATCH
“Resveratrol, the polyphenol in red wine, improves survival on a high-calorie diet, according to a landmark paper in Nature (Baur et al., 2006).”
the cited study was done in mice, flies, worms and yeast; neither this sentence nor its paragraph says so, so a reader takes it as a finding about people
Resveratrol improves health and survival of mice on a high-calorie diet
x NOT SUPPORTED SPECIES_MISMATCH
“The same work found that resveratrol increases insulin sensitivity and improves motor function, which is why longevity clinics now sell it by the bottle.”
the cited study was done in mice, flies, worms and yeast; neither this sentence nor its paragraph says so, so a reader takes it as a finding about people
Resveratrol improves health and survival of mice on a high-calorie diet
relative-risk.html
2 claims · 0 supported · 2 not supported · 0 not checked · 2 omission(s) · exit 1
x NOT SUPPORTED RISK_MAGNITUDE_OVERSTATED
“Aggressive blood pressure targets halve your heart risk”
the article says 50% lower ('halve'); the largest same-direction estimate in the source is hazard ratio 0.73, which is 27%. That is 1.85x smaller than the article's figure
All-cause mortality was also significantly lower in the intensive-treatment group (hazard ratio, 0.73; 95% CI, 0.60 to 0.90; P=0.003).
x NOT SUPPORTED RISK_MAGNITUDE_OVERSTATED
“Pushing systolic blood pressure below 120 halves your risk of a heart attack or stroke, a landmark trial found (SPRINT, NEJM 2015).”
the article says 50% lower ('halves'); the largest same-direction estimate in the source is hazard ratio 0.73, which is 27%. That is 1.85x smaller than the article's figure
All-cause mortality was also significantly lower in the intensive-treatment group (hazard ratio, 0.73; 95% CI, 0.60 to 0.90; P=0.003).
unchecked.html
4 claims · 0 supported · 0 not supported · 4 not checked · 0 omission(s) · exit 2
? NOT CHECKED NO_CITATION
“Magnesium glycinate before bed improves sleep quality within a week.”
this sentence cites no study, so there is nothing to check it against. That is not the same as the claim being wrong
? NOT CHECKED FETCH_FAILED
“A separate paper reports that magnesium supplements improve sleep quality in older adults (2026 study).”
the source pmid:41697399 could not be retrieved: ConnectionError: source lookups are offline (pass --network to allow them). The claim is unexamined, not disproved
? NOT CHECKED SOURCE_NOT_FOUND
“A daily walk reduces the risk of dementia by a third, researchers reported last year (journal record).”
the citation 10.9999/journal-of-invented-results.2024.00777 did not resolve to any record in Europe PMC, so the claim was never compared against anything
? NOT CHECKED NO_ABSTRACT
“Vitamin D deficiency causes winter depression in northern latitudes (review).”
the record for 31124288 exists but carries no abstract text, which is normal for paywalled and editorial content; only the full text could answer this
well-sourced.html
4 claims · 4 supported · 0 not supported · 0 not checked · 0 omission(s) · exit 0
= SUPPORTED SUPPORTED_BY_SOURCE
“In a randomized trial of 9,361 adults at high cardiovascular risk, a systolic target below 120 mm Hg lowered the rate of serious cardiovascular events from 2.19% to 1.65% per year over a median of 3.26 years (SPRINT, NEJM 2015).”
the cited source states this, and none of the named failure modes fired
Conclusions: Among patients at high risk for cardiovascular events but without diabetes, targeting a systolic blood pressure of less than 120 mm Hg, as compared with less than 140 mm Hg, resulted in lower rates of fatal and nonfatal major cardiovascular events and death from any cause, although significantly higher rates of some adverse events were observed in the intensive-treatment group.
= SUPPORTED SUPPORTED_BY_SOURCE
“In a cohort of 229,119 men and 173,141 women, coffee drinking was associated with lower mortality, and the authors wrote that whether the finding is causal cannot be determined from their data (Freedman et al., 2012).”
the cited source states this, and none of the named failure modes fired
Methods: We examined the association of coffee drinking with subsequent total and cause-specific mortality among 229,119 men and 173,141 women in the National Institutes of Health-AARP Diet and Health Study who were 50 to 71 years of age at baseline.
= SUPPORTED SUPPORTED_BY_SOURCE
“Resveratrol improved survival in middle-aged mice fed a high-calorie diet (Baur et al., 2006).”
the cited source states this, and none of the named failure modes fired
Resveratrol improves health and survival of mice on a high-calorie diet
= SUPPORTED SUPPORTED_BY_SOURCE
“In twelve trained male sprinters, beetroot juice taken 150 minutes before testing did not improve 60 m or 100 m sprint times (trial).”
the cited source states this, and none of the named failure modes fired
Acute beetroot juice ingestion fails to improve sprint performance and neuromuscular function in trained male sprinters: a randomized, double-blind, placebo-controlled study
What this tool cannot do
- It reads abstracts, not full texts. A study whose abstract omits its sample size is reported as “sample size not stated in the abstract”, never as a study with no sample size.
- It cannot read a paywalled paper, and says so rather than guessing.
- Claim extraction is a rule engine. A health claim phrased in a way the rules do not recognise is never judged at all, and its absence from the report is not a pass.
- Attribution is by proximity, with two narrow inheritance rules. A sentence whose citation sits three paragraphs away is reported as uncited.
- It judges whether the article matches the study. It has no view on whether the study is any good.
Sources resolved through Europe PMC. Not medical advice.