Exploring Speech-to-Speech Technology and Voice Actor Engagement

Anne welcomes Oz Krakowski from DeepDub, a company specializing in dubbing and voiceover end-to-end localization. Oz and Anne discuss the evolving technology and the importance of protecting the integrity and earnings of professional voice actors. They look at the ethical landscape of AI, the significance of artist compensation, and the transformative effect technology is having on voiceover work. With the rise of deepfakes and synthetic voice replication casting shadows of concern, they discuss the pressing need to protect voice artists' identities. They also navigate the intricacies of consent and compensation in voiceover AI models. This episode is a call for trust and clarity in the dynamic dance between technology and the voice talents that bring authenticity to AI-generated content. 00:01 - Intro (Announcement) It's time to take your business to the next level, the boss level. These are the premier business owner strategies and successes being utilized by the industry's top talent today. Rock your business like a boss, a VEO boss. Now let's welcome your host, Anne Ganguzza. 

00:20 - Anne (Host) Hey everyone, welcome to the VEO Boss podcast. I'm your host, Ann Ganguzza, and I am truly excited to be here with a very special guest, Krakowski. Oz is an experienced executive with a rich background in business and technology, especially in the entertainment industry. His current role at DeepDub showcases his deep involvement in the realm of media and film, where he leverages innovative technology to enhance the industry's landscape. His extensive knowledge and insights have made him a sought after figure in film and media conferences. Oz, thank you so much for joining us today. We're so happy to have you here. 

00:57 - Tom (Co-host) Hi Anne, so pleased to be here and thank you for inviting me and looking forward to our discussion. 

01:02 - Anne (Host) Yeah, absolutely. I'm excited to talk to you about the technology. So, first and foremost, for our listeners, tell us a little bit about DeepDub and your role there and your particular experience in working with voice actors. 

01:16 - Tom (Co-host) Absolutely so. Deepdub is a company that is focusing on AI, or generative AI, I should say solutions for audiovisual content, with an aim to democratize the ability to globalize content. We started by focusing on entertainment content, working with big studios small and large, actually and localizing and providing them the capabilities to use synthetic voices and different generative AI tools in order to localize their content. Specifically for me, I'm based in Dallas in the United States. I am responsible for business development, partnerships, strategic accounts, been with the company almost from the beginning. The company started in 2019, so we're pretty young in the world of, in the traditional world of localization. However, in the world of AI, we're among the first. 

02:06 - Anne (Host) You're veterans, you're veterans. 

02:08 - Tom (Co-host) Exactly, exactly. I always joke that it's like dog years every quarter it's like another leap forward in technology and advancements. We were there before. Generative AI was a common phrase on everyone's discussion. Absolutely this is generally about DeepDub DeepDub is really committed to the entertainment industry and asked about how we work with voice actors. There is a lot and I'm sure we're going to unpack it here in the discussion. 

02:33 - Anne (Host) Oh yeah, obviously, as you know, being a voice actor myself and you having worked with voice actors in the industry, I'm sure you understand the concerns of, basically, actors and artists in the creative fields and their concerns about AI. I think that it's important that all of us, voice actors included, we educate ourselves on this evolving technology to figure out how we can work with it, because I don't think we're going to stop it. Number one I've certainly seen lots of evidence of that in the past few years. I mean, it has just been breakneck speed in developing synthetic voices. From your perspective, what would you say are the major concerns surrounding the usage of AI in entertainment? In the entertainment industry? 

03:21 - Tom (Co-host) Absolutely. I followed your podcast and I see the work that you've done on AI. I truly respect the will and intention to actually be aware of it. Like you said, I think it's also acknowledged today that this is pretty much unstoppable. The question is, how do we get in front of it? How do we actually address the concerns? How do we work together and not necessarily try to do anything that is one-sided? 

03:45 In the end, everyone has fear when it comes to change and changing landscape or changing conditions, especially when it comes to a person level. People have their fears and a deep that we're trying to address them instead of not trying to behind anything. We're addressing them by working with the industry, by adhering to common practices, to the most recent laws or ethical codes that have been published. One of the things that we've just recently announced about a month ago I think it's almost like a month and a half ago, just when the actor strike was concluded we announced what we call the Voice Actors Royalty Program, which is a clear step forward in terms of giving voice actors, specifically professional voice actors, the ability to get compensated for the use of their voice in AI-powered projects. 

04:40 That specifically addresses professional voice actors. We did not open it to the public. In fact, when you join the program, we actually ask for proof that you are a professional voice actor. Once we get that proof, we vet it. We actually make sure that this is the right that you are indeed who you claim you are, or with the specific credentials. 

05:00 - Anne (Host) You have to do that today, in this age of AI. 

05:04 - Tom (Co-host) Because essentially I have a microphone, I can claim that I'm a voice actor. It's not that it's not going to be open ever to everyone, but the intention initially is to ease voice actors that we are not going to use their voice without their full consent and without them getting compensated for it. I think that's very important. 

05:23 - Anne (Host) Yes, I agree with you 100%. I think one of the biggest issues now is because it's the Wild West out there I like to use that term all the time in the voice acting industry because it's always been like a Wild West of rates. With new media and digital media, it's the Wild West of synthetic voices in AI. Now there are so many technical things that arise in terms of how can voice actors protect their voice? There are… a lot of companies out there drafting up these ethical policies and agreements that they post on their websites, and I think there is a bigger issue at stake where I think that it's wonderful when companies can proclaim and create policies and ethical guidelines, but what's out there? Who's out there that's enforcing them? Number one, I think really at a federal level or a global scale, there needs to be laws and regulations on that, and I think that's where we're going to be playing catch up, for maybe I don't know, this is my guesstimation the next 10 years, maybe more. What are your thoughts on that? 

06:23 - Tom (Co-host) You know what? You're absolutely right and this is, I think, the key challenge that we have, that the legal system is so slow to progress. However, you know, technologically we're advancing fast. So what happens? It leaves the playing field, you know, open for interpretation, and this is where it really matters. 

06:42 - Intro (Announcement) And, just like you, said this is the wild west. 

06:44 - Tom (Co-host) Exactly, and this is where it's really important when you partner with a company to actually do the background work, make sure that you're working with a company that really cares about not only the output, but also how you achieve that output, because today you can achieve that output in many ways, but it's really important who are the people, what's their track record, how do they do things, and not only what do they achieve in the end and why? Because there is no global enforcement right now. 

07:12 I think it's going to get there eventually. There's no way around it. 

07:15 - Anne (Host) It's going to have to get there, but it's just going to yeah. 

07:18 But what happens in the interim? I think that's the biggest thing. I have so many questions for you and one of them, of course, I wanted to like wait a little bit, but obviously I'm sure you've heard of the groundbreaking agreements between SAG-AFTRA and Replica, another AI company, in terms of working with actors and voice actors to protect them, and I think that that was a step in a positive direction, toward companies wanting to be more accountable, to not just themselves and not just their own ethical guidelines. Like I know, there's so many wonderful organizations out there trying to create guidelines and research, and I think that that's great, but I also think that is a step towards I would say, at least a company solidifying that they are serious about protecting the rights of creatives and actors and voice actors. What are your thoughts on that? Is that something that you have been looking into or a place where you might go in the future? 

08:09 - Tom (Co-host) Absolutely. First of all, sag-aftra is in the United States, so I think it's a great step forward for sure Definitely protects and sets the tone for unions and guilds worldwide. We have been discussing for some time with the BFFS, which is like the SAG-AFTRA or the similar union in Germany. I, just for the sake of discussion, I was invited by them last June to talk about AI in the film industry at the Munich Film Festival and we had a very interactive panel discussion there at that event, and I think that goes back to how we started the discussion today, saying that we need to acknowledge that this is coming and let's be in front of it and have the dialogue and talk about how we do things, and I think that agreement with SAG-AFTRA is important and an important step forward. 

08:57 I believe we're going to see similar things happening also worldwide, not only in the United States, and I think also from SAG-AFTRA perspective, it will be their intention to have that type of agreement with as many, I would say, ai companies create some kind of standard in a way. 

09:11 - Anne (Host) Yeah, let's talk a little bit about in the technical aspect of things, because, again, I don't work in AI, but I try to learn as much as I possibly can so that I can make educated decisions about how I can go about evolving with the industry as we move forward with the rapid advancements of AI technology, specifically generative AI I mean, is it discussed in any other realm anymore as generative AI? How can artists protect themselves from their unauthorized usage of their voice or their likeness or whatever that might be? I mean, right now there has to be technology that can be put in place so that we might be able to find out if our voice is being used without our permission or our likeness. 

09:52 - Tom (Co-host) The reality, I would say, is that technology today allows that to be abused. I mean, that's the reality. And it goes back to what I said before If you're a very famous voice, talent or it could be a talent or just a voice, talent or talent of your voice. 

10:10 I mean, or a celebrity, for example, and there's enough information about your voice. We actually today we don't even need a lot. Then your voice can be out there. We don't need much. You know it's like a sentence. Basically, your voice could be out there. The thing is, you know what happens. If it does. You can take it, for example, into places where we see non-entertainment related voices, like political people. You know people from politics where you know their voices can be replicated and can be abused. Actually, you know, especially if we're going into an election year in the United States deep fakes are very real very real, exactly I will say that. 

10:49 - Anne (Host) That gives me hope, though, that it is a political season, that maybe it'll get more government level. There may be more action taken quicker because of that. I don't know, that's just. That's just my speculation. 

11:01 - Intro (Announcement) But in reality. 

11:02 - Anne (Host) So, and even this podcast. I mean gosh, I've had a podcast for seven years. I mean, not only have I done thousands of auditions and given my voice freely, after being paid for a job, to my clients, there are so many ways that my voice has probably already been put into a database somehow. And I think that, technologically speaking, how does deep dub? First of all, how do you create voices? I guess the creation of the voice and then usage of the voice. How do you protect your actors, your voice actors, in both of those cases? Can their voices be used for training other models, and what's the usage requirements? 

11:40 - Tom (Co-host) Yeah, and that goes back basically I'm connecting my two answers before basically of first, there is no way really, I mean, everyone can use your voice because your voice specifically, you have that podcast with hundreds of hours it can be used without your knowledge. And then it combined it with partnering with the right company that actually you can trust, that is committed to working with you in terms of doing things in a legal way, in an ethical way and also compensation wise. So this is, if you combine it together. Now it goes back to the way we had deep dub require voices. 

12:16 There are two ways for us to acquire voices. First, with non-professional voice actors, where we actually pay them for the voice. We approach people that do not use the voice as their profession and ask them in a very consent way to contribute their voice. They're getting paid for it and once their voice is in, we actually use those voices as the baseline for our bank voices. They're not necessarily going to hear their voice in the outcome, because those voices get transformed internally and mixed with other voices, essentially, and the output is different. But this is the baseline for how to acquire voices. 

12:54 - Anne (Host) Yeah, so they're being paid for their voice being a training model then, to generate or create new voices, and then do they get paid if that new voice is being used since they were part of the training model or no? 

13:06 - Tom (Co-host) No, not necessarily. No, they're contributing their voice and, again, they're not professional voice actors. They're not getting paid. They're getting paid for the work that they did and they're not gonna see their voice in the output. Also, it's not gonna be their exact voice yet. 

13:19 - Anne (Host) Okay. So then now the case of a voice actor. If a voice actor wants to be on your roster. 

13:24 - Tom (Co-host) So in a voice actor, it's a different case. A voice actor, their voice is actually going to be in the output and they're going to be paid in the same way as they're getting paid today for doing the voice acting, only that they don't need to be in the studio. 

13:40 They could pretty much be anywhere else or be involved in a different job. So, essentially, you know, they're basically going to contribute their voice. We're not going to use their voice for training. We're not going to offer their voice as a bank voice. It's going to be offered as a voice that is essentially a royalty based or not royalty based, but a professional voice actors voice, which, every time this voice is being used for a project, they're going to be compensated for it. Now, mind you, like I said, it's very similar to the way their voice actors are getting paid today. When you participate in a production, you're getting paid for the participation. You're not getting paid for every time this is being broadcasted. 

14:22 - Anne (Host) Well, it depends on the type of job. I mean, if it were broadcast media, yes, you would get paid each and every time it gets broadcasted. 

14:29 - Tom (Co-host) So yeah, so it depends exactly in the end, and we're talking about the similar level of compensation as if you were doing the actual work, only now it's without attending the studio. 

14:39 - Anne (Host) Sure, sure. Now, how does that work in terms of when the voice gets chosen, is the actor then contacted, and then are they privy to how the voice is being used in terms of is it only for dubbing or could it be for other purposes as well? I mean, I may or may not want my synthetic voice, if it sounds like me, to be represented in a movie that might be something that I wouldn't consider myself wanting to be in you know what I mean or a production that maybe would be saying something that I wouldn't particularly want my voice reference to, for people to recognize and say, oh, and then assume that I would be of the same opinion. 

15:17 - Tom (Co-host) Yeah, listen, it's a great question. I will say that this is a place where that constantly evolves, because there is no enforcement, because the laws, there is no real legal framework. We're kind of like swimming in an open ocean and trying to define the land. Basically, for us right now, the idea is that when you do this, you're concerning to specific types of work. There is right now we didn't really set it up in that way, but the intention is really to give you the ability to try to actually vet yes or no based on types of work, ahead of basically your voice being used. 

15:50 So when you sign up to have the ability to actually say alright, this type of genre is I don't want to be in, but essentially we have a producer of the work that sits down. In a similar way that it's done in a real production, a producer will sit down and is casting voices from a list of voices. They can choose a bank voice or they can use a professional voice. If they use a professional voice, in the end there's an output saying this specific voice actor, their voice, has been chosen. It's not in the intermediary output, but the final output has a professional voice, artist voice in it. Then they're going to get compensated for it. 

16:26 - Anne (Host) Got it. Is it on a project by project basis? The payment, the compensation, in terms of how am I compensated? If I can get more in depth, because you know, what's so interesting is that I love you telling me that I'm being compensated. Now the question is okay, so what is that royalty share? What is that percentage? And is it varying depending on the project? Or is that something that I would have any input into negotiating with you? Or is that something and I realize because obviously you have created that synthetic voice and you're the one generating the files that yes, there is a certain percentage? Wise, that is obviously it's your work in generating that voice. Is it something that the voice actor can negotiate or is it you have just across the board? This is the royalty fee and it does it very based on, or some actors more, let's say in demand is one more of a custom voice than others. 

17:16 - Tom (Co-host) Yes, yes, yes and yes, yes, it can be negotiated. Okay, yes, some are more in demand and basically it's a yes on all of them. There's like a baseline you can negotiate if you have the ability to negotiate it. So yeah, overall, the answer is yes. 

17:31 - Anne (Host) What's the turnaround time? Just out of curiosity, if I would say that you're working on a production and I'm going to assume that these would be films that you're working with in terms of the amount of turnaround time and what this means for, let's say, the dubbing industry, like what is the advantage of using the synthetic voices and also I assume it's not just the voicing in order to make this? I guess a good experience for the consumer. You're also doing video effects as well, is that true? 

18:00 - Tom (Co-host) or we're not necessarily focused much on the video side, although we can. We figure we first address the challenge with the voices themselves and then move on to the video. That's coming up as well, the ability to actually change the video, to do a perfect lip sync, but at this point, you know, focusing on the audio itself. So there's like multiple ways of actually creating a production to localize a content. And let's step away for a second from a film, let's talk about a show, because the challenges with the show are, I would say to some extent are bigger, because there's just more content. And especially, what we've noticed is that the challenge becomes substantial, especially when we're talking about a big volume. You know when we're trying to scale it. And why is that? Because now everything becomes when you're working traditionally, everything becomes, you know, sequential, it's all serialized and it takes a lot of time, a lot of effort. You know you need a lot of resources. Think about a show that has I don't know 10 episodes, especially again, if we're talking about catalog content. It could be a show that has I don't know 10 seasons and now localizing it would take a lot of time, would take more than a year. And the thing is that without technology, this project becomes irrelevant, meaning it wouldn't be localized at all. So what happens is that we are suddenly enabling and quite frankly, our first focus is to try to address those because there's so much demand for localizing even sometimes older titles that have new markets and have never been localized, and it's impossible always. So what we do is we're enabling to do this very quick. When I say very quick, I mean we've recently dubbed the voiceover. So there is like difference between voiceover or you know, if you address with scripted, but we've done 100 episodes of 30 minutes in about six weeks for a customer, which is pretty fast. Another project that we've done was eight seasons of a show Again, it's catalog content, but it's still high quality scripted content, french to English that we dubbed 85 episodes in four months, which is without technology. It's just impossible to do it. The speed, or you're just taking the quality, really, really low, yeah. So we're looking at it as an enabler, not necessarily. And when you look at it this way, by the way, I have to mention to say that you should actually consider that not as a something that takes jobs, because those jobs would not have been there at all, because our customers would not dub it traditionally otherwise because the turnaround, time and the costs. 

20:32 So what happens in our process? It's never fully automatic. There's always people involved, whether a voice actor, sometimes too, a dubbing director. There's a studio sometimes involved in the process. There are linguists, we have translators. There's a lot of curation. That is done because, like you said, we care about the output. We really is done in a way that will actually be in line with the quality standards for streaming, for example. We have to align on that level of quality, otherwise, you know, we don't have a product. So that's very important. This is where we bring in people to be involved in the production itself. 

21:08 - Anne (Host) So I assume that after you are using the synthetic voice for the localization, for whatever it might be, you then have a little bit of post production on that, because I know that with the advances in technology things are really great. But how much are you having to also then additionally work on that to kind of get the emotion, Because of course people are all about it's the emotion that's lacking in the synthetic voices. And so what sort of work is involved these days after the synthetic voice is applied to bring that emotion and to bring that scene to life? 

21:40 - Tom (Co-host) Well, first, I must say that there is a lot of work that is done before we even apply the synthetic voices because, again, when we look at it, we look at it from an holistic point of view. At least the deep dove were not only a voice company where an end to end dubbing or localization house. We do the transcription, we do the translation, we have professional translators involved, together with the machine translation, to actually create the adaptation that is specific for that content, whether that's only referencing the cultural aspects or even addressing lip sync related issues, for adaptation related issues. Then we can record in two ways, whether we are using voice conversion or speech to speech. The same way I've seen some of the other guests you had here on the podcast talked about it, so we use that in a similar way or we also have the text to speech option. 

22:30 We recently launched our emotive text to speech that allows us to control the emotions of the output of the text to speech, which is also something pretty new and I would say to some extent mind blowing, because it gives us the ability to simulate a performance to some extent right on the output, right, right, we create those voices, whether in this method or the other, then we also mix. We always have a sound engineer at the end of the process where they will be able to take that and if we need to create a 5.1 mix, master the output, deliver it. There is some level of mixing that is also done using AI. That can be done automatically, but we always, always have a person in the process to curate and make sure that the output is in the right quality summits that we're aiming for. 

23:20 - Anne (Host) Just my own experience with speech to speech and understanding that speech to speech is not necessarily quicker in one respect, you know what I mean, because there's still an actor that's involved for that source acting. But I can absolutely see that the technology to probably put this together and make it realistic is mind-blowing to me and I actually wish I could see the process, just so I could know even more about it. But until that happens, talk to me about how voice actors can get involved, and I guess do you call it being on the roster, being on your roster? And again, you said there's a vetting process. What's involved if a voice actor is interested in having their voice with your company? 

23:59 - Tom (Co-host) So, first of all, like I said, we are already working with voice actors in different regions to provide the performance when we use the speech-to-speech. So we've done this over thousands of hours already. But if voice actors professional voice actors want to join our royalty program, it's as simple as going on our website and signing up to the royalty program. We're in a slow process of bringing people in. 

24:23 We're not rushing into it just because we are trying to vet everyone in and trying also not to create an oversaturation, Also on our end we're still a small company, so it does create some burden on us to actually make sure to vet everyone, but we're trying to do this in a very clear way that everyone is on the same page, there's no misunderstandings, and make sure that once we have someone on board, they know exactly what the process is. Go on the website, click the button, join the program, be part of this change. 

24:55 - Anne (Host) So one of my last questions is kind of a more generalized question about companies in general, because you are one of the handful of companies that I have spoken to that are actively speaking out and saying that you're supporting and wanting fair compensation for the actors and the creatives involved. So, from an organizational level, from a business standpoint, how can companies that develop and utilize AI act responsibly and manage the IP rights and concerns and ensure that they're respecting the rights of both human creators and AI generated content? 

25:26 - Tom (Co-host) I think it starts from the ground up. When you build your platforms, when you build your technology, you have to think of it from the ground up. If you did the majority of the way not thinking about, it is going to be very difficult to now reshuffle everything and now decide oh what? Now I want to be, go ethical and go legal, because you've already built, some from the beginning. 

25:49 Exactly, and this is the way we looked at it. This is one thing. The second thing is have dialogues, have communication. You know, listen. Part of the things, the reasons that I went to that Munich Film Festival and actually had several discussions with different units is, first of all, listen, empathize, try to understand the other side, try to understand how we can come up with solutions to actually address those concerns and not necessarily go on the highway and don't stop. So I would say this is the second thing. And the third thing is adapt. The landscape continuously evolves. We're just at the beginning of it in terms of the legal frameworks that are being put in place. So be able to adapt and adjust according to those changes. I think all three are important. 

26:32 - Anne (Host) What do you see coming in the future, not only for deep dove and yourself, but the future of AI technology. How do you see it evolving even more? 

26:40 - Tom (Co-host) Listen, I don't know if you noticed, but Riverside offers AI transcription. 

26:46 - Anne (Host) Yes absolutely, absolutely. My whole podcast is probably using AI. I mean, I use it to generate show notes and video clips, so absolutely. 

26:56 - Tom (Co-host) Yeah. So I think in the end, what I think it's going to get to, to a point where AI is going to be part of our lives in a very seamless way. Right now we are very judgmental about it, we're thinking about oh that's AI, and we're nitpicking on everything. But when it becomes seamless, you know it's just like. You know, cell phones in the beginning were bucky and huge and you had to carry a bag for it and it was. You had to actually think about it. Today, it's obvious that you go out of your house with a cell phone. You wouldn't do it otherwise. 

27:30 I think at some point, looking to the future, maybe a few years from now, ai is going to be more seamless, more integrated in many ways that are not necessarily trivial to us Even today. The simple ones are like transcription right, I mean, you wouldn't put someone right now to go through the podcast to transcribe it. When AI can do 95% of the work pretty good and the rest of the 5% you can do on your own, it becomes manageable. At the same time, I think that there's still going to be a place for us human beings in the process, basically responsible for the creativity. I don't think a lot of the fear is like AI is going to take our place. It's going to take over the world. Yes, absolutely, skynet is taking over. I think that eventually, what we're going to see is that it's just going to change the way we approach things more curating, more directing and guiding the AI, rather than trying to do it around. 

28:23 You know, for me, for example, today I want to write a formal email. I'll just go to chat GPD. It's not that I cannot do it. I can't do it myself and I've been doing it for you know, for years myself. But chat GPD, if I just give it a few point of reference, it'll give me something in seconds. And now I'm going to take it not as it is. I'm going to make a lot of changes in it and make it my own. Yeah, but I have the baseline and I think that's an indication of how it's going to be in the future in many other aspects of our lives. 

28:52 - Anne (Host) Yeah, I agree, and I think that as we progress forward, as it becomes more seamless hopefully it also becomes more regulation takes place so that we can be protected, so that it's not being used without our permission or knowledge and AI for good. 

29:06 I'm a believer, I want AI for good and I have seen where I think AI is so beneficial in so many ways and it's just a little bit scary to see it when it's not being used in great ways. But I press on and I think bosses out there, we need to educate ourselves so that we can evolve along with it and use it for positive. Use it for good. And, Oz, I want to thank you so much for joining me today. It's just been a pleasure talking with you and thank you so much for sharing your wisdom with us. 

29:34 - Tom (Co-host) Absolutely. Thank you so much for having me and I look forward to future podcasts. 

29:39 - Anne (Host) Absolutely All right. Bosses, here's your chance to use your voice, okay, not only possibly to do dubbing, but also to make an immediate difference in our world and give back to the communities that give to you. Visit 100voiceswhocareorg to commit and big shout out to our sponsor, ipdtl. You too can connect and network as humans and bosses, as I just did with Oz. Find out more at IPDTLcom. You guys have an amazing week and we'll see you next week. Thank you. 

30:08 - Intro (Announcement) Join us next week for another edition of VO Boss with your host and Gangusa, and take your business to the next level. Sign up for our mailing list at vobosscom and receive exclusive content, industry revolutionizing tips and strategies and new ways to rock your business like a boss. Redistribution with permission. Coast to coast connectivity via IPDTL.