OTTAWA — The Canada Revenue Agency paid $18 million for a chatbot the auditor general says gave her team the wrong answer 66 per cent of the time.
In February 2020, then-National Revenue Minister Diane Lebouthillier
she promised would help taxpayers better navigate the labyrinth of Canada’s sprawling tax code:
.
The chatbot could help answer general tax filing questions but was still “learning about the CRA,”
.
Fast forward five years and Auditor General Karen Hogan suggested that the chatbot that was present on 13 CRA web pages still has a lot of learning to do.
“Charlie’s responses tended to be brief, offering limited context and minimal additional information,” reads Hogan’s scathing report on
CRA’s call centres published in October
.
“We found that Charlie provided accurate answers in only 2 out of the 6 questions we asked it, while the other public web-based conversational artificial intelligence tools answered 5 out of 6 questions accurately.”
The finding stunned Liberal MP Anthony Housefather, who asked Hogan at the time “how is it that the government’s tool specialized for the CRA is worse than the average tool that anyone else can access anywhere?”
While reviews of Charlie the Chatbot at its inception suggested
the tool was somewhat unimpressive
, the cost to develop and maintain it were not.
New documents tabled in Parliament
this week reveal that taxpayers paid over $18 million since fiscal year 2018-2019 to develop and operate Charlie the Chatbot, with the bulk of spending beginning with its wide-scale launch in 2021-2022.
The agency says the bulk of that cost is salaries ($13.67 million), though that doesn’t include costs related to employee benefits and travel if necessary for the project. Another $3.21 million was spent on IT consultants for the project.
“Obviously, it’s a lot of taxpayer money,” said Conservative national revenue critic Gérard Deltell. “We have to recognize that 33 per cent, it’s not satisfactory at all.”
Over its nearly six years of existence, people have started over seven million “conversations” with Charlie and asked over 18 million questions, the agency said.
The document also confirms that the CRA’s chatbot has struggled to provide consistently accurate information.
The agency said there was a period where the measured “accuracy threshold” of the tool was 70 per cent, suggesting it offered inaccurate responses to the CRA’s testers 30 per cent of the time.
In November, the agency upgraded Charlie to a generative AI chatbot (closer to ChatGPT) that could respond to more questions. The agency says pre-release testing of AI Charlie suggested an accuracy rate of “approximately 90%.”
But “the exact number of questions the Generative Artificial Intelligence chatbot has answered correctly cannot be precisely determined without conducting a comprehensive review of all interaction transcripts,” the CRA added.
The data provided by the tax agency to Parliamentarians also suggests users come to Charlie with a very wide range of questions.
This year, Charlie was asked hundreds of thousands of questions on individual taxes in the months leading up to the April 30 filing deadline. Between February and April, it was also asked over 430,000 questions about the agency’s “electronic services.”
Charlie also received thousands of interactions this year about business income tax, benefits and credits, updating personal information, tax balances, payments and how to contact the agency.
In 2024, Charlie received nearly 180,000 interactions categorized as “chit chat.”
On Thursday, the CRA released the results of its “100-day service improvement plan” demanded by Finance Minister François-Philippe Champagne.
The agency said it doubled its responsiveness during the fall thanks in part to rehiring or extending the contracts of over 1,200 call centre employees and improving some digital services including Charlie the Chatbot.
“The GenAI chatbot beta can now answer a wider range of questions, including more complex ones for businesses, such as tax credits and compliance information,”
.
A spokesperson for the CRA did not answer questions by deadline about the cost of the chatbot at a time when developing such AI-powered tools is increasingly cheaper thanks to proliferating large-language model tools (such as ChatGPT).
National Post asked Charlie the Chatbot to explain why it cost $18 million to create and maintain, but the chatbot demurred.
“The context provided does not include specific information about the $18 million cost to create and maintain the CRA’s GenAI chatbot beta or any other system,” it responded while advising to contact the CRA.
National Post
cnardi@postmedia.com
Our website is the place for the latest breaking news, exclusive scoops, longreads and provocative commentary. Please bookmark nationalpost.com and sign up for our politics newsletter, First Reading, here.
