What’s behind the TikTok accounts using AI-generated versions of real Latino journalists?
A network of nearly 90 TikTok accounts has been using artificial intelligence to create fake versions of high-profile Spanish-language journalists and spread falsehoods online for potential financial gain.
Over a third of the accounts used AI-generated versions of Jorge Ramos, one of the best-known Latino journalists in the United States, to front fabricated news stories. One of them featured an AI avatar of Ramos falsely claiming that President Donald Trump’s son Barron Trump stormed into the United Nations to denounce the deportation of his mother, first lady Melania Trump.
“I never said that,” Ramos himself said in Spanish last month when he debunked the false narrative in a TikTok video posted on the account of this new independent news program. Ramos launched the show, “Así Veo las Cosas,” on social media this year following his exit from Univision in December after nearly 40 years at the network.
“There are things that are impossible to stop, and we can’t stop artificial intelligence right now,” Ramos said in his video. “There are tons of videos of me where I’m supposedly saying things I have never said.”
The accounts point to the challenge of stopping or controlling the surge in fake images and misinformation as AI technology advances and is increasingly used by those who want to spread false information online.
Alexios Mantzarlis, director of the Security, Trust, and Safety Initiative at Cornell Tech, Cornell University’s graduate campus in New York City, found 88 TikTok accounts that routinely used AI-generated versions of Ramos and other Latino news anchors from the Spanish-language networks Telemundo and Televisa to spread misinformation online targeting Spanish-speaking audiences in the United States.
NBC News reviewed the contents of the 88 accounts before TikTok shut them down after it learned of Mantzarlis’ findings.
Most of the 88 accounts were created this year and used AI avatars of Ramos, Noticias Telemundo and NBC News anchor José Díaz-Balart and Televisa anchor Enrique Acevedo. (Telemundo and NBC News are owned by NBCUniversal, a subsidiary of Comcast Corp.)
Their AI avatars, some of which were more realistic than others, were used to front false stories about divisive topics such as immigration, as well as conspiracy theories about Trump, Jeffrey Epstein and Sean “Diddy” Combs.
The most recent videos posted by the now-deleted accounts had the AI avatars talk about a fabricated story of an orca attack that went viral and a nonexistent curfew for children based on a false law authorizing the abduction of children in the United States. The comments on a video about the false storyline fronted by Acevedo’s AI avatar showed that while some users seem to have identified the content as false, other expressed distress over it, suggesting they believed the misinformation being spread.
“These deepfakes hijack my voice, my image, and — more importantly — the trust I’ve built with audiences over the years. I’m leaning on transparency, calling them out publicly, but the scale of this threat is bigger than any one journalist,” Acevedo told Mantzarlis, who wrote about his findings on his newsletter, Indicator, last week.
A TikTok spokesperson told NBC News in a statement that the company “banned these accounts for violating our Community Guidelines and continue[s] to vigilantly protect our platform from harmful misinformation and deceptive AI-generated content.”
Mantzarlis said there are probably hundreds more such accounts on the platform.
He first began researching the trend more than six months ago. In March, Mantzarlis discovered a network of nearly 40 TikTok accounts posing as Telemundo and Univision that used AI-generated content and the voices of well-known professional journalists to spread misinformation about topics that tend to go viral on social media. The accounts went undetected for about a month before TikTok shut them down.
But the trends Mantzarlis found on TikTok have evolved as more social media platforms integrate AI tools into their apps, making it easier to generate credible AI avatars, he told NBC News.
Based on his research, Mantzarlis said the creators behind such TikTok accounts are constantly trying different ways to generate content that creates large viewership numbers to accumulate at least 10,000 followers — which is the minimum required to monetize videos under TikTok’s Creator Rewards Program.
The creators have “determined that sensationalist news in Spanish, targeting a U.S. audience, does numbers, so they’ll try to feed that niche,” he said.
That’s why some of them have even used AI-generated versions of non-native Spanish speakers — including a Brazilian journalist and comedians from “The Daily Show,” an American satirical TV program — to spread Spanish-language misinformation.
Mantzarlis said he found “very strong evidence” suggesting that such TikTok accounts are being built up to garner enough followers to monetize their videos. The monetized TikTok accounts are then sold to other people “who can change the topic and theme and find another niche” they can profit from.
Mantzarlis found an encrypted chat group managed by Brazilian TikTok creators who claimed to sell monetized social media accounts that came pre-loaded with AI-generated clickbait content. In it, he saw someone claim to be selling a monetized TikTok account named “Tv Telemundo” for 300 Brazilian reals, or about $55. The account had posted AI-generated news and religious content to gain 11,000 followers under the previous name. The account now shares AI-generated wellness content.
Marta Planells, Telemundo’s vice president of digital news and streaming, told NBC News that the network has been reporting TikTok accounts impersonating Telemundo and their anchors for over a year. Once the accounts are reported, Planells said, TikTok has been proactive in shutting them down. But when that happens, more accounts come up, she added.
Even after Mantzarlis published his research last week based on the initial sample of 88 TikTok accounts, he found six other accounts publishing misinformation fronted by AI avatars of real Latino journalists. TikTok also shut down those accounts.
TikTok did not tell NBC News whether any of the accounts Mantzarlis identified were part of the Creator Rewards Program.
TikTok claimed in a company report published this year to have proactively removed more than 94% of the content that it identified as violating its policies about AI-generated content and misinformation.
Despite the efforts to remove false content, Ramos still encouraged his followers on TikTok to remain “vigilant, because misinformation is everywhere.”
“There are tons and tons of fake videos that appear to be real,” he said. “This, of course, creates a lot of confusion.”