Analysis: China Grapples With Its Own Epidemic of Deepfake Porn and Fraud

Elon Musk’s AI company, xAI, has come under increasing global scrutiny following an incident where its AI chatbot, Grok, enabled users to create and share sexually explicit, AI-generated pictures of children and women on the social media platform X. This controversy has alarmed regulators, especially in China, where authorities have become concerned about the rapid advancement of generative AI technologies enabling lawbreakers to easily and cheaply create nonconsensual deepfake images for illegal purposes[para. 1].

The misuse of generative AI has become rampant in China. According to government data, as of June 2025, the number of generative AI users in China reached 515 million, representing an increase of 266 million since December 2024. Young people are particularly active users of such technology in the country[para. 2][para. 3]. One high-profile case involved Hong Kong actor Louis Koo, whose voice and likeness were synthetically used in a fraudulent promotional video for an online gambling platform. His management was compelled to issue a statement denying his involvement, drawing attention to the risks of deepfake technology in fraud[para. 4].

Cybersecurity experts note that deepfake technology is widely used in emerging forms of telecom fraud since AI-generated videos are increasingly realistic and can learn nuanced human behaviors, making detection by the naked eye very difficult. Deepfake pornography, especially targeting minors, is also a growing issue: a survey by the Internet Society of China reported over 40% of respondents experiencing harassment by such content[para. 5][para. 6].

While the basic strategies of scams have remained consistent, they have shifted from online forums to short-video platforms, where much of the illegal content is now created with AI. For example, scammers generate fake video ads for legitimate financial products and upload them to video platforms, which then direct individuals to fraudulent lending schemes. Romance scams, known as “pig-butchering,” often use AI-generated adult content and celebrity face-swapping to attract victims[para. 7][para. 8][para. 9].

AI has also enabled a gray market where fraudsters defeat facial recognition, open counterfeit accounts, obtain loans, and impersonate people for blackmail. Local generative AI models in China are usually equipped with compliance mechanisms, adding content labels to flag AI-generated material. However, these safeguards mainly apply to major platforms with large user bases; individual developers using overseas, open-source models evade such regulations, complicating law enforcement efforts[para. 10][para. 11][para. 12][para. 13].

In September, Chinese authorities highlighted a case where a Zhejiang-based company's app was removed from stores after failing cybersecurity assessments for its deepfake upload service[para. 14]. Since mid-2025, major Chinese video platforms like Douyin, Kuaishou, and Bilibili have reported increases in illegal AI-generated content. Douyin alone removed over 380,000 videos in 2025 for violating pornography rules, with other platforms enacting similar cleanups[para. 15][para. 16].

AI-generated fake news exacerbates public anxiety and spreads quickly. Police have detained individuals who used AI to fabricate sensational fake news, such as a video of a deadly tunnel collapse, solely to attract online traffic[para. 17]. Because AI lowers barriers for such crimes, risk control specialists advocate for proactive detection systems to preempt illegal activity. Developing AI capable of identifying fake product ads or recognizing materials used in illegal AI videos is seen as a vital response[para. 18][para. 19].

Legislation is another front: in May 2025, U.S. President Donald Trump signed the Take It Down Act, which criminalizes the nonconsensual publication of intimate images, including AI-generated deepfakes. Offenders can face up to three years in prison, a move inspired in part by the viral spread of obscene AI-generated images of pop star Taylor Swift on social platform X a year earlier. China may consider similar legal frameworks to combat AI-enabled crimes[para. 20][para. 21][para. 22].