An autonomous large-language-model workflow identified documented cognitive concerns in clinical notes with high specificity but moderate sensitivity in a retrospective Mass General Brigham study. The experiment shows how routine records might support screening, while also showing why the system is not ready to stand in for clinical assessment.
The researchers analyzed 3,338 notes from 200 patients. The notes, recorded from 2016 through 2018, included clinic visits, histories and physicals, progress notes and discharge summaries. Clinician chart review from an earlier study supplied the reference labels for whether each patient's record contained a cognitive concern.
The Study Screened for Concerns, Not Alzheimer's Disease
The model answered a narrow question: did the available notes indicate any cognitive concern? A positive classification could reflect documented memory problems, confusion, behavioral changes or another reason for assessment. It did not determine the biological cause and did not distinguish Alzheimer's disease from other dementias, medication effects, delirium, depression or other conditions.
That boundary changes how the result should be read. The workflow may help identify a chart that deserves attention, but a flag is not a diagnosis. History, examination, validated cognitive testing and appropriate laboratory or imaging evaluation remain necessary to determine whether impairment is present and what may be causing it.
Five Agents Refined the Screening Instructions
The team built a workflow around the open-weight Llama 3.1 model. Five specialized agents examined classification errors and proposed revisions to the instructions used to screen notes. Their roles focused on tasks such as improving sensitivity, improving specificity and synthesizing the suggested changes.
Researchers compared that autonomous optimization process with prompts refined by clinical experts and with a static lexicon-based natural-language-processing baseline. The point of the agentic system was not that five independent clinicians examined a patient. It was that several computational roles collaborated to tune a classification prompt without human input during that optimization loop.
Performance Changed in the More Realistic Dataset
The development dataset contained 2,228 notes from 100 patients and was deliberately balanced: half of the patients had chart-reviewed cognitive concerns. In that setting, the final agentic workflow reached 91% sensitivity and 95% specificity after expert re-adjudication.
The independent validation set contained 1,110 notes from another 100 patients, with cognitive concerns present in 33%. There, sensitivity fell to 62% while specificity rose to 98%. The F1 score was 0.74 and accuracy was 88%. Because most validation patients were negative, accuracy alone would make performance look stronger than the missed-positive rate permits.
In practical terms, 62% sensitivity means the workflow failed to flag a meaningful share of records labeled positive by the final reference standard. High specificity means it produced few false positives in that dataset. A screening system must be judged on both sides of that trade-off, especially when a missed concern can delay assessment.
Re-Adjudication Exposed Problems in Both Labels and the Model
Independent expert review examined cases in which the autonomous workflow and the initial chart labels disagreed. In the validation set, experts concluded that seven of 16 apparent false negatives, or 44%, reflected defensible AI judgments rather than genuine model misses.
The remaining errors were informative. Some cognitive concerns appeared only in problem lists without supporting narrative. The system also missed clinical indicators including aphasia and limited health insight, and some agent outputs contained contradictory signals. The exercise improved the reference labels, but it did not erase the model's sensitivity limitation.
The Evidence Came From One Health System
The cohort was predominantly White and non-Hispanic, and all records came from Mass General Brigham. Documentation style, patient language, population characteristics and the prevalence of cognitive concerns can differ across institutions. The authors said external validation could not be performed because an equivalent chart-reviewed dataset was unavailable.
The workflow also saw clinical notes, while human reviewers could use the complete electronic health record. Notes may omit caregiver observations or early symptoms, and a problem-list entry without narrative can be difficult for a note-only system to interpret. Testing at multiple institutions and across more diverse racial, ethnic and linguistic groups is therefore a safety requirement, not optional polish.
Local Deployment Could Protect Data, but Governance Still Matters
The researchers used an open-weight model that can run inside hospital infrastructure, and Mass General Brigham said patient data need not be sent to an external cloud service. That architecture can reduce one privacy risk, but it does not answer every governance question. Health systems would still need access controls, audit logs, retention rules and a clear policy for who receives an alert.
They would also need to show clinicians why a record was flagged and what follow-up is available. An unexplained warning can be ignored or overtrusted. A useful decision support system must connect evidence in the chart to a defined clinical pathway without presenting its classification as a medical conclusion.
The Missed-Case Rate Is the Deployment Test
The study's strongest contribution is its transparent stress test. A system that looked highly sensitive in a balanced development set fell to 62% sensitivity when the positive share changed, even while specificity remained high. That is precisely the kind of shift a health system must detect before clinical use.
Automating chart review may help surface concerns that disappear across years of fragmented notes. But scale is not the same as safety, and cognitive-concern screening is not Alzheimer's diagnosis. Until external validation shows reliable performance in the population where the tool will run, the honest role for this system is experimental decision support. The standard should be the patients it correctly routes to care and the concerns it fails to see, not the sophistication of the agents behind the screen.