Connect with us

Accounting

Which generative AI model did best on the CPA exam? Depends on the section

Published

on

ChatGPT is no longer the only large language model to pass the CPA exam.

After ChatGPT 3.5 initially bombed the CPA exam and then version 4.0 passed, it does remain the top performer overall. However, like any human accountant, it has its strengths and weaknesses.

These were part of the findings of a recent paper from Case Western Reserve University and accounting automation solutions provider AIgency. The researchers systematically evaluated the performance of Google Gemini, ChatGPT-4, Claude, Mixtral and Llama-2b on multiple-choice questions from CPA test preparation tools.

Overall, they found that ChatGPT-4 scored the best, with Claude 3-opus coming in a close second, followed by Google Gemini Advanced, then Mixtral-8x7b-32768. Llama 2B-70b-4096 did the worst.

Source: William Zacher Jr. & Sanmukh Kuppannagari

However, as the results show, not every model did uniformly well on all sections. ChatGPT, while a strong performer overall, was especially good on the BAR section for business analysis and reporting. Meanwhile, although its weakest point is REG, the regulatory area that is mostly devoted to tax regulations, it did better on this section of the exam than any other model. Claude was the best performer in the AUD section on auditing and attestation. While its weakest point was FAR, the section on financial accounting and reporting, even there its performance was second only to ChatGPT. Gemini was the second strongest performer on the BAR section, but did not do so well on REG. Mixtral, overall, had decent enough scores compared to a human but would only pass BAR, making it a mediocre player compared to its peers. Llama was the only one that would not pass any section, and it did especially poorly on REG. It was also the only one that did worse than a human. The average score for human test takers on REG was 59.19%, according to the paper.

“The study revealed that while some LLMs have made significant advances in mimicking the complex decision-making skills required for CPA exams, there remains variability in performance across different sections of the test,” said the paper. “This variability underlines the importance of tailored training and specialization in developing LLMs for professional applications such as the CPA exams.”

To perform the test, the researchers drew their multiple choice questions from the Becker CPA test preparation suite. Google Gemini, Claude and ChatGPT-4 were accessed via their online platforms. Mixtral and Llama-2b models were accessed through the Groq platform, an advanced computational infrastructure for high-speed AI processing. The questions were directly copied and pasted into the AI platforms from Becker’s test preparation material without any additional prompting or modification to ensure each AI model received the questions in their original form as they would appear in a CPA exam context.

Becker’s platform randomized the questions in batches of 15 questions, which the research said further mitigated potential selection bias. The tester, responsible for inputting the questions into the AI models, deliberately refrained from reading or evaluating the questions beforehand to prevent any unconscious bias in the prompting process. For each question, the tester selected the AI model’s first response marked as “correct,” irrespective of any variations in the explanations or outputs provided by different models.

Each AI model was subjected to each multiple choice section of the CPA test three times, allowing for a comprehensive assessment of its performance across multiple attempts. The criterion for determining an AI model’s success in this study was achieving a passing score, defined as an average score of 75 or higher, on any given section.

The researchers said the data indicates there is no one universal model for all tasks, so it is important to use the right model for the right applications. For example, the paper concluded that ChatGPT is “the only real option for zero-shot BAR automation,” as “no other model came close to its performance, and it had a relatively narrow variance,” meaning that ChatGPT-4 could be used to help with automated financial statement preparation or additional forecasting. On the other hand, the researchers said Claude was probably better on auditing-related tasks, which the paper said “is a solid indication that it can be used for fraud detection and internal control validation.”

“It is apparent from the results that there is no clear-cut winner,” the researchers concluded. “Most companies utilizing AI to perform financial administration functions should use a software infrastructure that allows them to use multiple task-dependent AI models.”

However, the researchers did recommend that “model selection for AI in an applied accounting setting should avoid Llama-2B, which performed worse than any other model in every section.”

Continue Reading

Accounting

XcelLabs launches to help accountants use AI

Published

on

Jody Padar, an author and speaker known as “The Radical CPA,” and Katie Tolin, a growth strategist for CPAs, together launched a training and technology platform called XcelLabs.

XcelLabs provides solutions to help accountants use artificial technology fluently and strategically. The Pennsylvania Institute of CPAs and CPA Crossings joined with Padar and Tolin as strategic partners and investors.

“To reinvent the profession, we must start by training the professional who can then transform their firms,” Padar said in a statement. “By equipping people with data and insights that help them see things differently, they can provide better advice to their clients and firm.”

Padar-Jody- new 2019

Jody Padar

The platform includes XcelLabs Academy, a series of educational online courses on the basics of AI, being a better advisor, leadership and practice management; Navi, a proprietary tool that uses AI to help accountants turn unstructured data like emails, phone calls and meetings into insights; and training and consulting services. These offerings are currently in beta testing.

“Accountants know they need to be more advisory, but not everyone can figure out how to do it,” Tolin said in a statement. “Couple that with the fact that AI will be doing a lot of the lower-level work accountants do today, and we need to create that next level advisor now. By showing accountants how to unlock patterns in their actions and turn client conversations into emotionally intelligent advice, we can create the accounting professional of the future.”

Tolin-Katie-CPA Growth Guides

Katie Tolin

“AI is transforming how CPAs work, and XcelLabs is focused on helping the profession evolve with it,” PICPA CEO Jennifer Cryder said in a statement. “At PICPA, we’re proud to support a mission that aligns so closely with ours: empowering firms to use AI not just for efficiency, but to drive growth, value and long-term relevance.”

Continue Reading

Accounting

Accounting is changing, and the world can’t wait until 2026

Published

on

The accountant the world urgently needs has evolved far beyond the traditional role we recognized just a few years ago. 

The transformation of the accounting profession is not merely an anticipated change; it is a pressing reality that is currently shaping business decisions, academic programs and the expected contributions of professionals. Yet, in many areas, accounting education stubbornly clings to outdated, overly technical models that fail to connect with the actual demands of the market. We must confront a critical question: If we continue to train accountants solely to file tax reports, are we truly equipping them for the challenges of today’s world? 

This shift in mindset extends beyond individual countries or educational systems; it is a global movement. The recent announcement of the CIMA/CGMA 2026 syllabus has made it unmistakably clear: merely knowing how to post journal entries is insufficient. Today’s accountants are required to interpret the landscape, anticipate risks and act with strategic awareness. Critical thinking, sustainable finance, technology and human behavior are not just supplementary topics; they are essential components in the education of any professional seeking to remain relevant. 

The CIMA/CGMA proposal for 2026 is not just a curriculum update; it is a powerful manifesto. This new program positions analytical thinking, strategic business partnering and technology application at the core of accounting education. It unequivocally highlights sustainability, aligning with IFRS S1 and S2, and expands the accountant’s responsibilities beyond mere numbers to encompass conscious leadership, environmental impact and corporate governance. 

The current changes in the accounting profession underscore an urgent shift in expectations from both educators and employers. Today, companies of all sizes and industries demand accountants who can do far more than interpret balance sheets. They expect professionals who grasp the deeper context behind the numbers, identify inconsistencies, anticipate potential issues before they escalate into losses, and act decisively as a bridge between data and decision making. 

To meet these expectations, a radical mindset shift is essential. There are firms still operating on autopilot, mindlessly repeating tasks with minimal critical analysis. Likewise, many academic programs continue to treat accounting as purely a technical discipline, disregarding the vital elements of reflection, strategy and behavioral insight. This outdated approach creates a significant mismatch. While the world forges ahead, parts of the accounting profession remain stuck in the past. 

The consequences of this shift are already becoming evident. The demand for compliance, transparency and sustainability now applies not only to large corporations but also to small and mid-sized businesses. Many of these organizations rely on professionals ill-equipped to drive the necessary changes, putting both business performance and the reputation of the profession at risk. 

The positive news is that accountants who are ready to thrive in this new era do not necessarily need additional degrees. What they truly need is a commitment to awareness, a dedication to continuous learning, and the courage to step beyond their comfort zones. The future of accounting is here, and it is firmly rooted in analytical, strategic and human-oriented perspectives. The 2026 curriculum is a clear indication of the changes underway. Those who fail to think critically and holistically will be left behind. 

In contrast, accountants who see the big picture, understand the ripple effects of their decisions, and actively contribute to the financial and ethical health of organizations will undeniably remain indispensable, anywhere in the world.

Continue Reading

Accounting

Republicans push Musk aside as Trump tax bill barrels forward

Published

on

Congressional Republicans are siding with Donald Trump in the messy divorce between the president and Elon Musk, an optimistic sign for eventual passage of a tax cut bill at the root of the two billionaires’ public feud.

Lawmakers are largely taking their cues from Trump and sticking by the $3 trillion bill at the center of the White House’s economic agenda. Musk, the biggest political donor of the 2024 cycle, has threatened to help primary anyone who votes for the legislation, but lawmakers are betting that staying in the president’s good graces is the safer path to political survival.

“The tax bill is not in jeopardy. We are going to deliver on that,” House Speaker Mike Johnson told reporters on Friday.

“I’ll tell you what — do not doubt, don’t second guess and do not challenge the President of the United States Donald Trump,” he added. “He is the leader of the party. He’s the most consequential political figure of our time.”

A fight between Trump and Musk exploded into public view this week. The sparring started with the tech titan calling the president’s tax bill a “disgusting abomination,” but quickly escalated to more personal attacks and Trump threatening to cancel all federal contracts and subsidies to Musk’s companies, such as Tesla Inc. and SpaceX which have benefitted from government ties.

Republicans on Capitol Hill, who had —  until recently — publicly embraced Musk, said they weren’t swayed by the billionaire’s criticism that the bill cost too much. Lawmakers have refuted official estimates of the package, saying that the tax cuts for households, small businesses and politically important groups — including hospitality and hourly workers — will generate enough economic growth to offset the price tag.

“I don’t tell my friend Elon, I don’t argue with him about how to build rockets, and I wish he wouldn’t argue with me about how to craft legislation and pass it,” Johnson told CNBC earlier Friday.

House Budget Committee Chair Jodey Arrington told reporters that House lawmakers are focused on working with the Senate as it revises the bill to make sure the legislation has the political support in both chambers to make it to Trump’s desk for his signature. 

“We move past the drama and we get the substance of what is needed to make the modest improvements that can be made,” he said.

House fiscal hawks said that they hadn’t changed their prior positions on the legislation based on Musk’s statements. They also said they agree with GOP leaders that there will be other chances to make further spending cuts outside the tax bill. 

Representative Tom McClintock, a fiscal conservative, said “the bill will pass because it has to pass,” adding that both Musk and Trump needed to calm down. “They both need to take a nap,” he said.

Even some of the House bill’s most vociferous critics appeared resigned to its passage. Kentucky Representative Thomas Massie, who voted against the House version, predicted that despite Musk’s objections, the Senate will make only small changes.

“The speaker is right about one thing. This barely passed the House. If they muck with it too much in the Senate, it may not pass the House again,” he said.

Trump is pressuring lawmakers to move at breakneck speed to pass the tax-cut bill, demanding they vote on the bill before the July 4 holiday. The president has been quick to blast critics of the bill — including calling Senator Rand Paul “crazy” for objecting to the inclusion of a debt ceiling increase in the package.

As the legislation worked its way through the House last month, Trump took to social media to criticize holdouts and invited undecided members to the White House to compel them to support the package. It passed by one vote.

Senate Majority Leader John Thune — who is planning to unveil his chamber’s version of the bill as soon as next week — said his timeline is unmoved by Musk. 

“We are already pretty far down the trail,” he said.

Continue Reading

Trending