Generative AI’s rapid advancement has reshaped the landscape of fraud, making it accessible to ordinary individuals with minimal effort and cost. What was once a domain requiring professional skills, equipment, and connections has now become a simple task achievable with a smartphone and basic prompts. This shift has impacted businesses, enterprises, and information ecosystems across sectors, as fraudulent activities ranging from fake product damage claims to AI-generated disinformation surge.
E-commerce Fraud: AI Fake Images for Refunds
The e-commerce sector has been among the first to feel the impact. In November 2024, Yu Jin, a seller of plush toys, encountered her first case of refund fraud using AI-generated fake images. A buyer requested a partial refund one week after receiving a plush toy, submitting a photo showing the item with supposed burn marks and dirty stains. When Yu’s customer service team rejected the claim citing artificial damage, the buyer escalated to the platform, which ruled in their favor, resulting in a 50-yuan refund.
Upon closer inspection, inconsistencies emerged: the “cracks” on the toy’s soft skirt resembled those of ceramic products, defying physical logic. Verification by AI professionals and tools confirmed the image was AI-modified, with overly regular textures inconsistent with real materials. Despite submitting evidence, Yu’s appeal was denied until media intervention prompted the platform to reimburse her—though the buyer retained both the toy and the refund.
Similar incidents multiplied during the “Double 11” shopping spree, with “AI fake images for refunds” trending on social media. On November 21, a keyboard seller identified an AI-generated fake image submitted by a buyer for a refund claim. The image reused the same composition, background, and lighting as the buyer’s positive review posted three days after receipt but added exaggerated damage to keyboard caps. Thanks to clear evidence of inconsistency, the platform rejected the refund request. Notably, the buyer was identified as a college student, highlighting how ordinary individuals without prior ties to fraud networks can now engage in such activities.
Disinformation: AI-Powered Rumors and Blackmail
Beyond e-commerce, AI has revolutionized disinformation campaigns. Qi Yun, a public relations professional with over a decade of experience, faced an AI-generated “black article” targeting a listed company he served. The article contained fabricated details—including a non-existent K-line chart in the chairman’s office and false statements attributed to a fictional CFO—yet was formatted to appear as legitimate news, sparking online discussions.
Combating such content proved arduous. Qi’s team spent two days preparing evidence, including official announcements and refutations of each false claim, to appeal for the article’s removal. “Proving a negative is inherently difficult,” Qi noted. “How do you prove a chart doesn’t exist in an office, or that a person never made a statement?” Legal action remained impractical for most companies due to limited resources and the volume of AI-generated disinformation.
AI has enabled industrial-scale disinformation production. In June 2024, a Chinese MCN agency was reported by CCTV News to have used AI tools to scrape online information, generate sensationalized fake news, and monetize through traffic. The agency produced 4,000 to 7,000 AI-generated articles daily, earning over 10,000 yuan per day. In another case, a man surnamed Li generated a vulgar pornographic video using AI, superimposing fake content onto footage of XPeng Motors’ auto show booth. The realistic integration of characters, lighting, and background misled many netizens before police confirmed it was AI-generated.
Platform Oversight Struggles to Keep Pace
The proliferation of AI fraud exposes gaps in platform moderation systems. Most platforms still rely on outdated verification methods such as simple image comparison and keyword tagging, which fail to detect sophisticated AI-generated content. For AI-generated images and videos, some platforms add small disclaimer labels, but text content—including fake news and reviews—remains largely unmarked and hard to identify.
Academic circles have also grappled with unreliable AI content detection tools, with high error rates in identifying AI-generated academic papers during the 2024 graduation season. As a result, high-quality, authentic content is increasingly overshadowed by low-cost, AI-generated content prioritized by algorithms for engagement.
The imbalance between low fraud costs and high countermeasures costs has shifted the burden to victims. Fraudsters face minimal risk—AI tools are free and accessible, and penalties for detection are rare—while businesses and individuals spend significant time and resources on verification and appeals. This dynamic has led some professionals like Qi to leave the PR industry, frustrated by the shift from fact-based debates to battling technical loopholes.
The New Reality of Accessible Fraud
Generative AI has democratized fraud by reducing entry barriers to near-zero. With simple prompts, anyone can generate convincing fake images, videos, or text in seconds. This accessibility, combined with low perceived risk, has blurred the line between “can” and “should,” prompting ordinary individuals to experiment with fraud out of curiosity or convenience.
Given the advancing trend of AI technology, the challenge of regulating its abusive use intensifies. The gap between AI’s fraud-enabling capabilities and platform moderation systems continues to widen, threatening trust in e-commerce, media, and public discourse. Addressing this requires technological innovations in AI content detection, updated platform policies, and collaborative efforts to raise awareness of AI fraud—all while balancing innovation with accountability in the digital age.
