Words mean more than what is set down on paper. It takes the human voice to infuse them with deeper meaning.
(Quoted by Maya Angelou)
For humans, their voices form a core part of their identity, allowing them to express thoughts and feelings. Within creative industries such as media and entertainment, it is often the case that the success of an artist is associated with how effectively they can use their voice in performances such as movies and TV series. On the other side of the spectrum, we find informative and educational purposes, where instructors or teachers utilize their voices to craft learning experiences that facilitate students in retaining and comprehending knowledge effectively. So precious and indispensable is it that the human voice has made its way to synthetic or generative media technology. This has opened up new opportunities for more seamless and scalable content creation, distribution, and localization.
For those who are not familiar with the concept of synthetic media, it refers to any type of media that is created or modified by Artificial Intelligence (AI). The methods commonly employed for generating replicated media involve deep learning and machine learning techniques. Synthetic voices fall under the umbrella of synthetic media, although you may have heard them referred to as 'deepfake voices' or 'voice cloning' in public discussions. The term 'deepfakes' can sound suspicious, creating a sense of deception. Consequently, many content owners and producers prefer to use the term 'synthetic media' to avoid such connotations.
In the remainder of this article, you will be introduced to the following topics:
➊ A Brief Look at the Evolution of Media: Past, Present, and Future
➋ What Does Synthetic Voice Mean Within Synthetic Media?
➌ Potential Positive Application Areas of Synthetic Voices
➍ How AI-powered Dubbing Will Transform Content Localization Efforts
A Brief Look At The Evolution Of Media: Past, Present And Future
In the Synthetic Media Landscape 2020 report published by Samsung Next Ventures, media is observed to undergo three phases, each defined by the specific technology enabling content creation and distribution. In the old media phase (phase 1), a few publishers, broadcasters, and studios dominated the mass distribution through TV, print, and radio. The advent of new media (phase 2) brought about the democratization of content creation and distribution, facilitated by the widespread adoption of the internet and the proliferation of social media platforms. According to the same report, the future will be shaped by synthetic media, creating an ecosystem where every content creator and owner can access the means and resources needed to pursue their creative endeavors, thanks to AI-powered technologies.
This comprehensive report acknowledges AI technologies, deep learning, and synthetic productions as the catalyst for a revolutionary stage in the media landscape, attributing their profound impact to the empowering potential they offer to everyone, rather than a select few gatekeepers and behemoths. However, the report also highlights the challenges arising from the misuse and unethical applications of synthetic media, emphasizing the necessity for robust regulatory frameworks and corporate mechanisms to ensure the verification and truthfulness of content.
Synthetic or generative media encompasses various forms of content, including but not limited to music and sound synthesis, image synthesis, avatar synthesis, video synthesis as well as speech and voice synthesis. In this article, our primary focus is on synthetic voice or speech generated using AI, which is also referred to as voice cloning, replicated voices, or cloned voices. We will discuss potential positive use cases associated with synthetic voices. Subsequently, we will narrow our scope further to explore how AI-powered dubbing solutions are transforming content localization.
What Does Synthetic Voice Mean Within Synthetic Media?
Synthetic voice, also known as voice cloning, cloned voice, or deepfake voice, utilizes a person's recorded speech to generate a replica of their voice, allowing it to say words that the person never actually said. Initially, training AI models for synthetic voice generation was a time-consuming process, requiring hours of recorded speech from the voice owner. However, with advancements in neural networks, it is now possible to train AI using random recordings of a person's voice, with durations as short as a few minutes. AI, machine learning, and natural language processing form the core components of this efficient process, accelerating the generation of synthetic voices and enabling the creation of more lifelike and natural-sounding voices.
Potential Positive Application Areas of Synthetic Voices
As the capabilities of synthetic voices continue to advance, they are attracting increasing interest from content owners and distributors. Not only do corporations in creative industries utilize synthetic voices, but companies across various functions also leverage them to create a unique branding voice and serve as conversational assistants.
Some notable use cases include:
➡ Entertainment and media companies can use this technology to clone the voice of a well-known figure from the past, allowing them to produce films or documentaries directly narrated by that specific individual's voice.
➡ Celebrities, such as football players and movie stars, who face high demand from brands, especially during their busiest seasons, find voice cloning beneficial. It eliminates the need for them to spend long hours shooting scenes and traveling, while enabling brands and advertisers to run campaigns without delays or restrictions due to time and location.
➡ Synthetic voices are particularly useful for repetitive content, such as weather reports or sports updates, as they bring diversity to broadcasting and can scale up to meet high demand.
➡ Content creators, including those who develop educational online courses, can use their own cloned voices or someone else's AI-powered synthetic voice (with consent) to dub their existing content into other languages. This expands their outreach beyond local borders.
➡ Synthetic voice generation has also been instrumental in cases like that of Val Kilmer, who lost his voice due to cancer and benefited from voice cloning technology to regain his voice.
Source: LuckyStep/Shutterstock.com
These are just a few examples of the positive applications where AI-based synthetic voices have proven successful. As technology improves, it will be possible to create more iterations of replicated voices, resulting in even more natural-sounding, dynamic voices with human tonality and native-like qualities. Consequently, more applications will continue to emerge year after year. Notably, the use of artificial intelligence in localization solutions, particularly for dubbing content into other languages, stands out.
How AI Dubbing Will Transform Content Localization Efforts
Dubbing audiovisual content into other languages using cloud-based, AI-powered solutions is one of the use cases within synthetic media. In dubbing efforts, the goal is to create the most natural-sounding AI-powered voices, ensuring they are free from accent issues, overlaps, and synchronization problems. By 'natural sounding,' we mean a voice that sounds human-like rather than robotic, while also capturing the essence of the original acting.
Scholars from the International Institute of Information Technology, Hyderabad provide a comprehensive debate on the use of machine learning and artificial intelligence in translation. They identify three mechanisms that enable the generation of translated speech output from a given source speech input: 'speech recognition,' 'neural machine translation,' and 'speech synthesis modules.'
The system essentially comprises three key stages:
◘ Speech-To-Text AI Transcription (with minimal human post-editing expected).
◘ Text-To-Text Translation based on the AI-generated transcript (with minimal human post-editing expected).
◘ Text-To-Speech Dubbing using the AI-based translated transcript (with minimal human post-editing expected).
Reduced production costs and shortened delivery times are distinguishing features of this technology. With traditional dubbing methods, it typically takes one to three months to dub a feature-length film or documentary. However, with AI-powered dubbing, the process can be shortened by up to ten times. This transformational benefit addresses the shortage of talented voice artists that the fast-growing streaming and broadcasting media industry is currently facing.
The rise of global streaming platforms like Netflix, Hulu, as well as other content distribution channels such as FAST, ADVODs, and social media platforms like YouTube, has led to an increasing demand for subtitling and dubbing that humans alone cannot meet. By automating repetitive aspects of the process, AI frees up human resources, allowing them to allocate more time and mental energy towards editing and enhancing the AI-generated content to a higher standard.
Final Remarks
AI dubbing heralds the arrival of positively disruptive changes in the entertainment and media industry. At Ollang, we like to refer to it as 'AI-powered' for a reason: humans still play a vital role in the dubbing process. They contribute to creating synthetic voices, refining the outcomes, and adding the human tonality that cannot be achieved with AI alone.
We particularly recommend the use of AI dubbing for documentaries, e-learning materials, and videos with relatively lighter characterization and dialogue. An excellent example of this is a documentary project where Ollang's AI-powered dubbing tool was employed to voice over from French to English.
While AI-powered voicing may have limitations in certain genres like comedy or drama, end-to-end AI dubbing solutions managed and refined by humans can effectively utilize this technology.
We are prepared to embrace new challenges and handle your next localization project. By uploading your content to Olabs, you can expand your reach to a broader audience. If you have any questions or need further clarification, please refer to our FAQ section or feel free to contact support@ollang.com directly to discuss the specifics of your projects.