Older AI models show signs of cognitive decline, study shows — but not everyone is entirely convinced

AI Diagnostics: Do Chatbots show Signs of Cognitive Decline?

Artificial intelligence (AI) is rapidly changing the landscape of healthcare, with tools increasingly used for medical diagnoses. Their ability to quickly analyze medical histories, X-rays, and other datasets, often spotting anomalies before they are visible to the human eye, is revolutionizing the field.However, a new study published in the BMJ raises concerns about the reliability of AI in clinical settings, suggesting that these advanced technologies may exhibit cognitive decline with age, similar to humans.

Testing AI Cognition

Researchers tested publicly available large language models (LLMs) like ChatGPT, Sonnet, and Gemini using the Montreal Cognitive assessment (MoCA). This standardized test is commonly used by neurologists to assess cognitive function in humans, evaluating areas such as attention, memory, language, spatial skills, and executive function.

While the LLMs performed reasonably well in tasks related to naming, attention, language, and abstraction, they struggled considerably in visual/spatial skills and executive function. Interestingly, the older Gemini 1.0 LLM scored significantly lower (16 out of 30) compared to the latest version of ChatGPT (26 out of 30),leading researchers to speculate about a potential decline in cognitive abilities over time in AI.

Cautions and Criticisms

“These findings challenge the assumption that artificial intelligence will soon replace human doctors,” the study’s authors wrote. “as the cognitive impairment evident in leading chatbots may affect their reliability in medical diagnostics and undermine patients’ confidence.”

However, the study has faced criticism from other scientists who question its methodology and interpretation. Some argue that applying a human-centric test like MoCA to AI is flawed, as it doesn’t accurately reflect the way these systems process data. They contend that LLMs excel in tasks involving text processing and information retrieval,which are fundamentally different from the visuospatial and executive functions measured by the MoCA.

“The MoCA was designed to assess human cognition, including visuospatial reasoning and self-orientation — faculties that do not align with the text-based architecture of LLMs,” wrote Aya Awwad, research fellow at Mass General Hospital in Boston.

Lead author of the study, Roy Dayan, responded to the criticism, emphasizing that the study’s humorous framing in the Christmas edition of the BMJ should not overshadow its serious intent. He stressed that the goal was to highlight the limitations of AI in certain cognitive domains and encourage a more nuanced understanding of its capabilities in healthcare.

Looking Ahead

The debate surrounding AI’s cognitive abilities highlights the complexities of integrating these technologies into healthcare. While AI holds immense potential for improving diagnosis and treatment, it’s crucial to recognise its limitations. Further research is needed to understand how AI systems learn, evolve, and adapt over time. Ultimately, a collaborative approach, combining the strengths of human expertise and AI capabilities, will be essential for realizing the full benefits of AI in medicine.

Could the findings of this study, showing parallels between AI cognitive decline and aging humans, impact how we develop and implement AI in healthcare?

AI diagnostics: A Déjà Vu of Human Cognitive Decline?

Polly Newton, Archyde News Editor, sits down with Dr. Ada Sterling, a distinguished neurocognitive scientist, to discuss a recent study that raises intriguing parallels between AI cognitive decline and aging humans.

Revolutionizing Medicine,or a House of Cards?

Polly Newton (PN): Dr. Sterling, the use of AI in medicine is rapidly expanding. Yet, a study published in the BMJ sheds light on a captivating yet concerning similarity between AI and human cognitive function. Could you elaborate?

Dr. Ada Sterling (AS): Indeed, Polly. The study attempted to test various large language models (LLMs), such as ChatGPT and Gemini, using the MoCA, a human cognitive assessment tool. Interestingly, while these models performed reasonably well in certain tasks, they struggled in visuospatial skills and executive function. Moreover, the older Gemini 1.0 model scored significantly lower than the newer ChatGPT, raising speculations about AI cognitive decline over time.

Applying Human Tests to AI: A Flawed approach?

PN: Some scientists have criticized the use of MoCA on AI, arguing that it’s a human-centric test that doesn’t reflect how LLMs process data. Do you agree?

AS: While the MoCA was indeed designed for humans, it can still provide valuable insights when applied to AI. It helps us understand the limits of these systems’ capabilities, which are crucial for safe and effective AI integration into healthcare. However,you’re right; these findings should be interpreted with caution. We should aspire to understand AI not just by measuring its cognitive abilities, but also by studying its unique data processing strengths and weaknesses.

The Future of AI in Medicine: Collaboration over Replacement

PN: So, should we be worried about AI replacing human doctors any time soon?

AS: The goal shouldn’t be about replacing humans with AI, but rather harnessing AI’s potential to augment and enhance human capabilities.The study suggests that while AI can excel in information processing and pattern recognition, it might struggle with tasks that require human-like cognition. Thus, a collaborative approach, combining human expertise and AI capabilities, could lead to better patient outcomes.

PN: Thought-provoking indeed, Dr.Sterling. Your insights provide a fresh perspective on the intricacies of integrating AI into healthcare. Thank you for joining us today.

What do you think about this study’s findings? Should we rethink the role of AI in healthcare? Share your thoughts in the comments below.