Generative AI has improved to the point where it is now capable of producing fake documents realistic enough to fool automated systems, creating new opportunities for fraud and new challenges for those trying to prevent it. 

This new capacity came as part of OpenAI’s image generation update a few months ago, which dramatically increased the quality of AI-generated images, including financial documents. People found that, with just a simple prompt, they could produce extremely realistic looking receipts, invoices and other records, sometimes even adding wrinkles or smudges for extra verisimilitude. Very quickly people realized the fraud potential of such tools, as these images were found to fool even certain automated software systems. 

Anti-fraud professionals like Mason Wilder, research director with the Association of Certified Fraud Examiners, believe the issue is not so much the fake documents themselves but the fact that they can now be quickly and easily produced at an industrial scale. He noted that people have been forging documents since time immemorial, however doing so tended to need a lot of time, effort and expertise, which created a high bar for such activities. This meant that even if someone had thought about, say, inflating their expense reports with fake receipts, the effort required to do so was beyond what most were willing to do. 

But now people don’t need to edit things in Photoshop or alter text with whiteout. Instead they just need to describe what they need in detail, and an AI model will produce the requested file. 

“It opens the door for lazier fraudsters. You don’t even need to be sufficiently motivated or technically sophisticated to carry out a fraud scheme that 5–10 years ago would’ve required some level of technical sophistication and more motivation and time and energy. Now you can just do it in an afternoon pretty easily,” said Wilder. 

This is not just a theoretical problem, as AI-generated fakes are already being used in fraud schemes, such as the case of a Singaporean man who faked $16,000 of receipts—and this was even before the image generation updates. Further, according to Wilder, it’s not just receipts: people are also using generative AI to create realistic looking IDs, support false insurance claims, fraudulently apply for government benefits and more. This leads to schemes like one where an Italian man faked 2,600 boarding passes to exploit flight discounts offered by the Sicilian government.

While there is widespread agreement this is a problem, there is less when it comes to what exactly to do about it. Some have suggested using metadata to detect AI images, with certain vendors like T&E solutions provider Ramp updating their product to look for markers particular to generative AI systems. Once those markers are present, the software flags the receipt as a probable fake. 

“When we see that these markers are present, we have really high confidence of high accuracy to identify them as potentially AI-generated receipts,” said Ramp’s Dave Wieseneck in a previous article. “I was the first person to test it out as the person that owns our internal instance of Ramp and dog foods the heck out of our product.” 

David Zweighaft, a partner at forensic accounting firm RSZ Forensic Associates, said professionals in the field might take a similar approach. There are already ways to look at documents for evidence of alteration. While theoretically someone could strip out the metadata, he said that doing so, itself, creates new evidence of alteration. 

“We’ve got to move past the 2-d world we live in and look at the metadata, look at any traces that any electronic transactions or electronic modifications might leave. [We] may want to work with the software providers to come up with validation,” he said. 

He added that cases like these are exactly why people developed data forensics as a field. While the actual forensic work might be more difficult and complicated when dealing with AI, he felt the overall principles were sound. 

“This crisis is not new. Ever since computer-generated information has been used in litigation, it came up. … And that is where data forensics was invented—and that is where all of the legal defense work around data and making sure things were unchanged began. And now you have data validation and MD5, SHA-256, or MD64 hash algorithms to prove something was not changed from its original pristine state on the computer. This is just the latest iteration of that scenario,” added Zweighaft. 

Wilder, however, said that in order for data to become a foolproof way of verifying authenticity, there would need to be some sort of widely-adopted industry standard that mandates the inclusion of certain metadata (essentially, a watermark) in AI-generated images that can’t be removed. And even if that happened, he wasn’t sure how sustainable that technique would be in the long run. 

“As mainstream, institutional-type software providers agree to incorporate that into their services, there’s still a big issue: a lot of these LLMs and other AI models have been open-sourced at some point in the recent past. That means the underlying code is in the hands of whoever wants it, and they can build on top of it and make their own AI tools. So even if there is industry-wide adoption of some kind of tech standard like that, that is not going to really account for, you know, people who’ve built their own AI models. And there are a lot of really smart bad guys out there,” he said. 

While the immediate instinct for many would be to solve this problem with AI, Wilder was skeptical. Automated systems are easy to fool, and even if they’re powered by AI models, AI does not have the best track record when it comes to detecting AI. He pointed to a large number of cases where people put their own work through an AI detection solution only to find the software concluding it was done by computer. Overall, he felt the tools for generation were far outpacing the tools for detection, which makes them a poor choice for detecting AI-generated fakes. 

“You’ll have solutions providers telling people in the anti-fraud industry that, like, you can just use AI to solve this problem for you. And I would encourage people to exercise that professional skepticism in those contexts as well, because, you know, with emerging technologies, we’ve seen countless examples of people overstating the capabilities of AI tools. So I would encourage anti-fraud professionals to be really wary of the claims of solutions providers on the detection capabilities of their tools,” said Wilder. 

Instead, he felt professionals will need to start leaning on “more old fashioned controls” such as requiring everyone to use company credit cards that can be monitored, retrieving actual financial records versus screenshots (with the employee’s consent), and generally being more diligent in monitoring for anomalies and problematic patterns. He added that most companies can view what people do on their network, and so looking to see if someone’s Internet history literally shows them making the fake receipt can help too. And to account for external fraudsters, he recommended that contracts include a Right to Audit clause that lets them request official bank records from actual financial institutions to corroborate expenses. 

Todd McDonald, founder and CEO of financial intelligent software provider Valid8, however, felt that AI and automated systems must be part of the solution, even if it’s not as one generally imagines them. Recalling an exhaustive investigation into a Ponzi scheme that was done fully manually, he felt stepping away from automation was a bad idea. 

“Having to recreate the books and records for a Ponzi scheme, where there weren’t tools like the ones we’ve now built to validate things—at that time, we had to spend thousands of hours recreating the books and records from subpoenaed bank records—hundreds of thousands of transactions, over 12 years, across 20 entities. That was all manual, and it did not require the best of our skills and training. It was an unbelievably burdensome effort. We had to identify what had happened before we could even move on to what we could do about it. I didn’t have that luxury. I had to go through months of painstaking work just to get a data set I could trust before I could interrogate it and understand it,” he said. 

So while asking an AI “is this AI?” may not yield good results, this is far from the only option. Valid8 doesn’t look at a picture of a receipt and determine whether or not it is real but, rather, pulls actual records like bank and credit card statements or copies of deposit slips and checks, and uses that to verify discrepancies or duplications. This in mind, he himself is unconcerned with the AI’s ability to fake documentation, as his company concerns itself with the actual data. 

“It really comes back to the provenance of where you are getting the support for this documentation. At Valid8, we come with a specific point of view: bank statements don’t lie. They are a fundamental ground source of truth… There’s nothing immediate we’ve done as a result of the announcement or some of the new tech that is out there. It hasn’t changed things one bit from our roadmap to expand from using bank support evidence as a ground truth and being able to augment and enhance that with additional supporting documentation,” he said. 

However, he also noted that technology is only part of the solution. Having “highly trained humans” to actually interpret the data and understand the context is vital, as is training those humans to exercise professional skepticism and compliance, and checking to make sure those lessons were absorbed. There is still value in the old fashioned controls to which Wilder referred.  

“You should be setting up a culture of compliance, a clear and outlined code of conduct for what the expectations are regarding expense reports. You should set up a random audit methodology, and employees should know there are consequences for that. This is just good old blocking and tackling—someone is paying attention,” he said. 

George Barham, director of standards and professional guidance with the Institute for Internal Auditors, raised a similar point in that while it is unlikely people will step away from automated systems, they do need to be taught to take the outputs with a grain of salt and not blindly trust what the AI tells them. 

“I think the main thing is not completely relying on what the tools give you and being critical and looking at the results and asking questions or looking for trends. ‘gosh this cost really jumped over this year, what is going on?’ I also think if you look at a large number of items, it is still a good idea to take a couple and look at those annually so that won’t be a departure from how internal audits look at things, but I think you take what tech provides and what AI provides with a grain of salt,” he said. 

However, Barham was hesitant on any specific prescriptions for action, as every company is different and has different goals. So rather than outline what controls should be implemented in response to AI forgeries, he instead said it’s important that professionals sit down with managers and discuss what controls specific to the organization might be needed. 

“The biggest thing is making sure we’re having conversations … with management. Hopefully, they will do an annual risk assessment and maybe a quarterly mini-assessment. But you’d like to see some actions taking place from a risk assessment. So maybe that means adding or improving some of the controls in this elevated risk area. That could include more policies, more procedures, more controls, more reviews, more authentication methods when looking at receipts and understanding the source. So it falls to how the organization understands risk,” he said.