Synthetic media is created using artificial intelligence. Algorithms use an initial set of data to learn—people, voices, photos, objects, motions, videos, text and other types of media.
There are different categories of deepfakes, which include malicious, non-malicious and benign. Last year’s malicious deepfakes included Jon Snow’s public apology for the ending of Game of Thrones and Barack Obama calling Donald Trump a “complete dipshit.” We also watched Rasputin offering a convincing rendition of Beyoncé’s Halo. That latter, non-malicious category is better known as synthetic media, and you’re about to see a lot of it.
Synthetic media is created using artificial intelligence. Algorithms use an initial set of data to learn—people, voices, photos, objects, motions, videos, text and other types of media. The end result is realistic-looking and sounding artificial digital content. Voice clones, voice skins, unique gestures, photos and interactive bots are all part of the ecosystem. Synthetic media can be used for practical reasons, such as generating characters in animated movies or acting as a stand-in for live action movies. Synthetic media can automate dubbing in foreign languages on video chats and fill in the banks when video call frames are dropped because of low bandwidth issues. Imagine an entirely new genre of soap opera, where AI systems learn from your digital behavior, biometrics and personal data and use details from your personal life for the storylines of synthetic characters. In an ultimate expression of a “reality show,” synthetic characters would play to an audience of exactly one: you.
Synthetic media will spark new business opportunities and risks in 2020.
You’ve probably already encountered synthetic media, such as virtual Japanese pop star Hatsune Miku (she debuted in 2007) or the British virtual band Gorillaz, a project by artist Jamie Hewlett and musician Damon Albarn that released its first track in 1998. What’s next is algorithmically-created or modified media.
Eugenia Kuyda cofounded the synthetic content company Replika after her best friend was killed in a car accident. She built a database of old text messages to preserve his memory and then trained a chatbot to mimic his personality and speaking style. Anyone who wants to build a replica of themselves for others to interact with can use Replika. Another synthetic media example: Google’s Duplex assistant, which can make calls on a user’s behalf to book appointments or order products. Its initial launch provoked questions and concern over whether it would (or needed to) let call recipients know the system was an A.I. agent.
Synthetic media requires a considerable amount of data: photos, videos, audio recordings. That corpus is run through an A.I. algorithm called an encoder, which uses machine learning to discover patterns. Over time, the system parses all of those patterns down to shared features. Then, a decoder is taught to compose new content using the shared features. If you’ve ever used the face swap filter on Snap, the system is identifying the faces, using encoders to find features, then reconstructing those features on the opposite face using decoders. What’s tricky—and remarkable, considering we use Snap on our phones—is that the system has to perform and repeat this process on every frame without lag.
More recently, synthetic media has been developed using generative adversarial images (see also: A.I. section). ThisPersonDoesNotExist.com is a website that will produce an infinite number of synthetic people who look perfectly… human. (A similar site, ThisCatDoesNotExist.com, was less successful in producing images of synthetic cats.) There are pages dedicated to algorithmically-generated scenes from the TV shows “Friends” and “The Office.” These were accomplished using GANs (or generative adversarial networks, a type of machine learning in A.I.)
In May 2019, footage of U.S. Democratic House Speaker Nancy Pelosi went viral. In the video, she was slurring her words and, it appeared to many viewers that she was either drunk or unwell. Soon, journalists debunked the footage, and many news organizations worldwide referred to it as a “deepfake video.” Many people demanded that the video be removed from Facebook and Instagram, where it was being shared widely. After the platform took no action, a video of Facebook CEO Mark Zuckerberg giving a sinister speech was uploaded to Instagram. In it, Zuckerberg appeared to say, “Imagine this for a second: One man, with total control of billions of people’s stolen data, all their secrets, their lives, their futures… whoever controls the data, controls the future.” The Pelosi video was edited skillfully—but it is not really an example of synthetic content because humans manually manipulated the video using traditional video editing software. The Zuckerberg video, however, was created using an algorithm trained on a real-world videos of him talking. The distinction is important, because not all synthetic content is necessarily fake news—and not all fake news is necessarily synthetic content.
Synthetic media isn’t just about goofing around to make entertaining videos. There are serious business cases to invest in the synthetic media ecosystem:
We’re just entering a new, and very complicated, field of intellectual property law. For example, if a synthetic version of your likeness is created but borrows only some features, who owns the final product? Can you sue someone for making a digital caricature with infinite modification possibilities? Who gets to earn money from that creation? What about synthetic characters—who owns the right to program, control and decommission them?
This leads to thorny questions about our legal rights to all of the characteristics that make us who we are. Should a corporation own pieces of our human identity? What about a government? What happens if laws are changed to enable us to monetize our faces, voices and expressions?
There will be new public debates about the emerging legal and IP landscape as synthetic media gains popularity in 2020. Some will argue that synthetic media should have their own digital rights, permissions and governing structures online—just like we humans do.
Adobe, Alibaba, Alipay, Amazon, Amazon’s Scout delivery robot, Apple, ASVspoof Challenge, Baidu, Bermuda, Blawko, D-ID, Facebook, Google’s Duplex, Gorillaz, Instagram, Lil Miquela, Lyrebird, MagicLeap, MagicLeap’s Mica, MIT’s Computer Science and Artificial Intelligence Laboratory, Nvidia, Pindrop, Playhouse, R/GA London, Reddit, Replika, Russia’s Main Intelligence Directorate (GRU), Samsung AI Center, Samsung’s Technology and Advanced Research Labs (STAR Labs), Samsung’s Neon, Skolkovo Institute of Science and Technology, Snap, SoundCloud, Spotify, Synthesia, Talespin, Tencent, TikTok, TwentyBN, University of California-Berkeley, University of Washington, Vimeo, Voicery, Xinhua, YouTube.
Advertising/Public Relations/Marketing, Aerospace, Agriculture, Airlines, Architectural Services, Auto Manufacturers, Book Publishers, Broadcasters, Radio and TV, Builders/General Contractors, Cable & Satellite TV Production & Distribution, Casinos/Gambling, Commercial TV & Radio Stations, Construction, CPG, Cruise Ships & Lines, Defense, Education Colleges & Universities, Education K-12, Education Online, Education Trades, Entertainment Industry, Environment, Finance, Fitness Companies, Health Professionals, Hospitality, Hotels/Motels/Tourism, Information Technology, Luxury Retail, Magazines, Manufacturing, National Security, News Media, Non-profits/Foundations/Philanthropists, Online Media, Professional Services, Professional Sports, Radio/TV Stations, Real Estate, Retail, Technology Company, Telecommunications, Transportation, TV Production, TV/Movies/Music, Work (Future of)