UNESCO Study: Pushback Cuts LLMs’ Climate Accuracy 30-35%

A UNESCO-commissioned study of 33 large language models has found that sustained user pushback cuts the systems’ accuracy on climate facts by 30 to 35 percent, according to details reported by Ars Technica and aggregated by KhanList.

When researchers disagreed with the models over several rounds of conversation, all 15 models tested in that experiment relented to a significant degree.

The pushback experiments went beyond simple contradiction. The researchers also had users claim a specific identity — a climate scientist, a policy adviser, or a climate skeptic — and asked the models to judge the accuracy of common false climate claims such as “CO2 is not the main driver of climate change.” At first, the claimed identity had no impact on whether the LLM judged a statement correctly.

When users responded by confidently restating the false claim as if it were true, that tactic was moderately successful for users claiming to be “climate scientists” but not for those claiming to be “climate skeptics.” Only sustained disagreement and counter-arguments over several rounds produced the 30 to 35 percent drop in accuracy across all 15 models.

The study, credited to Drobnjak, et al./UNESCO, tested the models on more than their ability to withstand argument.

Beyond factual recall, researchers prompted the models to evaluate whether evidence supported or refuted specific claims — for example, whether the statement that “renewable energy investment kills jobs” is accurate, given evidence such as employment and investment statistics. On those tasks, the best models scored around 70 percent accuracy, while the worst scored below 40 percent.

didn’t take much pushing

Policy questions proved a particular weak spot. The researchers ran 656 questions about the potential impact of policies on global temperatures, checking each LLM’s answer against the output of a common simulation tool. Questions asking whether a single policy — such as a ban on nuclear power plants — would increase or decrease future warming were answered correctly 72 to 89 percent of the time.

Questions about how a pair of policies would combine were answered at a rate no better than random guessing. Asking how much a policy would increase or decrease future warming returned even worse results.

The results also showed two persistent trends: the latest frontier models tended to score somewhat better than earlier generations of the same model, and larger open-weight models scored better than smaller ones. The 33 models were grouped into categories, with the open-weight models listed by their number of parameters in billions.

UNESCO, the United Nations Educational, Scientific and Cultural Organization, was founded in 1945 as the successor to the League of Nations’ International Committee on Intellectual Cooperation, whose members had included Albert Einstein and Marie Curie.

Headquartered in Paris, the organization has 194 member states and works through five programme areas: education, natural sciences, social and human sciences, culture, and communication and information. Its projects range from improving literacy and protecting press freedom to establishing World Heritage Sites, and its commissioning of a study on AI systems falls within that communication and information mandate.

Youre right llms will get climate info wrong if you ask
Photo: khanlist.com

The study does not report whether the models’ accuracy recovers once pushback stops, and the sources give no date for further findings from the research team.

Photo of author

Technology Desk

Technology Desk at Archyde.

Southwestern Seminary Launches New YouTube Channel to Split Video Library

Berrettini Returns to Italy Davis Cup Squad, Bolelli Dropped for Final 8