AI chatbots unable to accurately summarise news, BBC finds

Four major artificial intelligence (AI) chatbots are inaccurately summarising news stories, according to research carried out by the BBC.
The BBC gave OpenAIâs ChatGPT, Microsoftâs Copilot, Googleâs Gemini and Perplexity AI content from the BBC website then asked them questions about the news.
It said the resulting answers contained âsignificant inaccuraciesâ and distortions.
In a blog, Deborah Turness, the CEO of BBC News and Current Affairs, said AI brought âendless opportunitiesâ but the companies developing the tools were âplaying with fire.â
âWe live in troubled times, and how long will it be before an AI-distorted headline causes significant real world harm?â, she asked.
The tech companies which own the chatbots have been approached for comment.
âPull backâ
In the study, the BBC asked ChatGPT, Copilot, Gemini and Perplexity to summarise 100 news stories and rated each answer.
It got journalists who were relevant experts in the subject of the article to rate the quality of answers from the AI assistants.
It found 51% of all AI answers to questions about the news were judged to have significant issues of some form.
Additionally, 19% of AI answers which cited BBC content introduced factual errors, such as incorrect factual statements, numbers and dates.
In her blog, Ms Turness said the BBC was seeking to âopen up a new conversation with AI tech providersâ so we can âwork together in partnership to find solutions.â
She called on the tech companies to âpull backâ their AI news summaries, as Apple did after complaints from the BBC that Apple Intelligence was misrepresenting news stories.
Some examples of inaccuracies found by the BBC included:
- Gemini incorrectly said the NHS did not recommend vaping as an aid to quit smoking
- ChatGPT and Copilot said Rishi Sunak and Nicola Sturgeon were still in office even after they had left
- Perplexity misquoted BBC News in a story about the Middle East, saying Iran initially showed ârestraintâ and described Israelâs actions as âaggressiveâ
In general, Microsoftâs Copilot and Googleâs Gemini had more significant issues than OpenAIâs ChatGPT and Perplexity, which counts Jeff Bezos as one of its investors.
Normally, the BBC blocks its content from AI chatbots, but it opened its website up for the duration of the tests in December 2024.
The report said that as well as containing factual inaccuracies, the chatbots âstruggled to differentiate between opinion and fact, editorialised, and often failed to include essential context.â
The BBCâs Programme Director for Generative AI, Pete Archer, said publishers âshould have control over whether and how their content is used and AI companies should show how assistants process news along with the scale and scope of errors and inaccuracies they produce.â