When it goes right, AI can seem like magic—all the busywork done for you, data shaped into clear insights, problems solved before you even notice. But when it goes wrong it can be a nightmare of wasted time and expensive mistakes. And, unfortunately, there are a lot of ways it can go wrong. But while it can be tempting to blame the software, most cases of AI error come down not to the bot but the user. 

One of the biggest issues, mentioned over and over, is that people simply do not double check the outputs of their AI models. While people are told over and over again that AI models can make mistakes, the majority of people generally do not take the time to confirm what AI is telling them. A McKinsey survey found that just 27% of respondents whose organizations use generative AI say that employees review all content created before it is used. A similar share says 20% or less of gen-AI-produced content is checked before use. And another study from trend analytics company ExplodingTopics found the problem was even more severe: Only 8% of people regularly bother to verify AI information, and 42.1% of web users have experienced inaccurate or misleading content in AI overviews. 

David Wood, an accounting professor at Brigham Young University who helped author an AI governance framework for organizations, said he himself has not seen many instances of significant AI errors, but one thing he does see a lot are people who simply forget to double-check model outputs and miss where the AI inserted a mistake. 

Vatcharachai – stock.adobe.com

“I think the number one error in terms of frequency is some type of small hallucination, and the user copies and pastes and doesn’t review. They don’t review the output. That is the biggest one. They go too fast without considering what it actually says. That’s, oof, 95 percent of errors I’ve seen,” he said. 

Jeff Seibert, founder and CEO of accounting solutions provider Digits, made a similar point. He noted that Digits’ goal as a company is to automate the preparation of the books. Not to review them, or verify them. That is up to the user. He stressed the importance of really making sure humans double-check what the AI is telling them. 

“That is the biggest risk: a business owner just blindly trusting the AI bookkeeping model. It may look correct, but might not follow the guidelines because it will be biased by what has been previously done in their books… Our guidance is that every business should still work with a firm or have someone qualified who can review and sign off on the finances,” said Seibert in an interview. 

Gina Montgomery, director of AI, automation and analytics at top 25 firm Armanino, said that the damage that comes from ignoring the human review element may not just be monetary but legal and reputational as well, as it could lead to firms submitting flawed deliverables just because the AI sounded confident. She mentioned how we’ve already seen large organizations turn in work with AI-generated errors and suffered for it. 

“That’s the exact kind of instance where you talk about human review: fabricated data, hallucinated citations, misapplied logic can reach regulators or investors before [the firm detects it] if you don’t have that layer built in,” she said. 

This type of risk grows as oversight weakens. She said firms should not apply a set it and forget it mindset when it comes to AI, particularly where it concerns automation. It’s important to clearly define what an AI is and is not allowed to do, but too many organizations ignore this control. 

“The most common missing control is a clear definition of what the AI is allowed to do. So installing an automation tool and assuming the guardrails are just built in, but don’t document approval thresholds or decision boundaries,” she said. An AI system without these controls operates without context and so becomes more likely to make errors.

But even if controls are built into the process, there is also the matter of getting people to follow them, as the phenomena of “shadow AI” (the unsanctioned use of AI) is growing in workplaces. EisnerAmper, a Top 25 Firm, recently conducted a study and found that a significant portion of professionals don’t tell their supervisors they’re using AI. While slightly more (22.4%) say they get permission first before using AI, almost as many (21.7%) have no such reservations; 22.2% either might or might not. Wood, the BYU professor, noted that this means a lot of the errors come down to the individual. 

“It’s not Walmart putting this into their AP process and it goes awry. If there’s mistakes, most will be from shadow AI or shadow IT. Someone is using it and not letting other people know. It’s not part of the regularly designed process,” he said. 

Ellen Choi, the founder and CEO of accounting AI consultancy Edgefield Group, noted that many large accounting firms already have strong control frameworks in place and so, like Wood, has not seen that much in the way of catastrophic AI failures. However, she did point out that there seems to be a consistent thread of people not necessarily knowing who is and is not using AI within the firm. 

“I’m not sure what it is but people don’t seem to want to talk about AI use with each other within the firm. … You get these meetings where I ask about AI use, and one person waxes poetic about how they love it and everyone else looks surprised. It happens all the time. It seems pretty obvious that when firms don’t know what their people are doing, this can have unintended consequences of not using AI in a way that is compliant according to the guardrails the company may have set up,” she said. 

For instance, she spoke of one firm where a junior associate put a client document into ChatGPT despite being explicitly warned against doing that specific thing. The only reason this person was caught was that the firm did happen to have IT controls that allowed them to detect it—else no one would have known. While the firm was able to work with OpenAI to get the documents removed from its training data, Choi said this is very rare and most commonly “it’s like ink and water. Once something has been added to the training data, it’s very, very difficult to isolate and remove it.” 

“That person did get fired. Did it have actual business consequences beyond the risk exposure, the bottom line financial impact? No. But I think this kind of stuff is definitely something firms should be vigilant about,” she said. 

Beyond expense and embarrassment, failing to implement proper controls over AI and keeping the human in the loop can serve to further degrade the usefulness of a firm’s AI model. This is because many models will actively learn from what the humans do, picking up the processes and procedures particular to that practice. Much like any human worker, if they’re given bad information, they’ll produce bad results, and if they’re not corrected by a supervisor, they’ll think that’s what they’re meant to be doing. 

“It learns the best practices of each accounting firm. Obviously, if you have a rogue accountant in your firm doing bad accounting, yes, the model could pick up some of those practices,” said Seibert from Digits, though he stressed that there is also a global model that can correct the local firm model if it acquires bad habits. 

Montgomery, from Armanino, talks about how letting mistakes into the training data can have cascading effects, which underscores the need for leaders to have independent verification layers before anything reaches the general ledger. 

“AI errors that reach the ledger could directly affect reported earnings or compliance status. The most frequent examples might be misclassifications, incorrect accruals or unauthorized payments. These are not code failures but governance failures. When AI is trained on inconsistent data or outdated coding logic, it can replicate past mistakes at scale. A misclassification error that a human might make one month could be repeated thousands of times automatically, which could certainly cause some big issues,” she said. 

To illustrate her point, she spoke about a medical organization client whose I would have made a very expensive error if not for last minute human intervention. The AI was attached to the procurement system, where it was responsible for ordering supplies as needed. Given a great degree of autonomy, the AI did most of the work itself. In this case, the AI had to order more gloves. Unfortunately, it had confused 20 boxes for 20 cases after misinterpreting a unit field. 

“That discrepancy was caught during manual review. So those validation points are not bottlenecks. A lot of people think about it that way, thinking ‘oh no, I’ve got to add a human to that.’ But it’s more like the brakes that make the automation safe,” she said. 

Controls also need to account for the types of AI used; if someone is using the wrong kind of model for something, it won’t matter how good the data is or how strong the guardrails, it won’t be able to do the job well. Using the right tool for the right job is important not just in AI but overall. Yet Seibert, from Digits, has seen too many people expecting large language models to do math when that’s not really their strong suit. He noted that Digits deliberately does not use LLMs to do any of the actual accounting work, relying instead on deterministic models and calculators to do the math, the results of which can then be communicated via LLM. 

“When you ask it a question we don’t want it to make up an answer. You can, in Digits, ask a question about finances, it can be ‘how much did we spend on marketing this year versus last year.’ Hand that to ChatGPT and it will literally make up an answer and the math may look right but may be subtly wrong since it’s not really doing math. … We have to go to extreme lengths to prevent our models from doing math. This is a common failure place. A lot of companies are not treating it that seriously. There will be subtle issues in the predictions because the models are hallucinating,” he said. 

But also, sometimes the right model does not remain the right model. Especially when someone licenses an AI model versus building their own, the company that controls it might make a new version or patch a current one. This can lead models to have wildly different behaviors even from small changes in its code. 

“You see it occasionally, where you’ve designed some kind of process with the API where the old model is not as good as the new one but they didn’t update to the newer model. Or the model changes and they don’t go back and test it,” he said, adding that it is not necessarily true that the most recent version is the best for your particular purpose. “People always expect 5 to be better than 4 but that’s not how generative AI works. It might work better at 95% [of things] but that 5% catches you.” 

Montgomery also talked about this risk, adding that often vendors won’t even provide notice that they’re changing the behavior of their AI model. 

“Contracts with vendors should have a certain level of transparency and notice on what they’re doing and you need a right to audit the AI’s behavior based on the way it is stated it is going to work…  Firms have to assess not only their own AI solutions or models, but also the controls of every provider that they depend on,” she said. 

This speaks to the fact that even if a firm’s own controls are sterling, there is still the matter of third parties. This is especially the case with AI agents which can act semi-autonomously. The issue with agents interacting with agents is that, by definition, a human is not involved. If every AI agent is using accurate information in the proper context, this is not such a big deal. But if an agent is acting on bad information, this can create a cascade that spread through an entire system. 

“As AI digital workers interact, there’s a growing risk of error propagation: one AI system can accept and reinforce another one’s incorrect output, and that is not good. … Without human validation, misinformation can spread faster than any individual could ever have done. So organizations have to design interaction protocols where AIs do not self validate. That’s where I think we are. Self validation between AIs is not a good idea. Any system to system exchange should include human confirmation or independent verification logic. Something like that has to happen. Human oversight being the last line of defense ensures accountability even when machines are collaborating,” she said. 

With all this in mind, Montgomery said firm leaders need to design their AI control structure with the same rigor as they would any internal control. 

“Every model should have a named owner, a review schedule, an audit trail. Accountability had to be specific, not collective. … I think the human relationship defines accountability for both the human side and the machine side. It’s easy to write really beautiful code. It’s just that the AI is going to execute exactly as it’s designed. So if you don’t have that validation in place, you’re in trouble,” she said.