It was a pleasure hosting the third annual IA Summit in person on October 2, 2024. Nearly 300 founders, builders, investors, and thought leaders across the AI community and over 30 speakers dove into everything from foundational AI models to real-world applications in enterprise and productivity. The day featured a series of fireside chats with industry leaders, panels with key AI and tech innovators, and interactive discussion groups. We’re excited to share the recording and transcript of the fireside chat on “Fakes, Deepfakes, and the Search for True” with Oren Etzioni, TrueMedia.org founder & former CEO of AI2, and Tom Standage, deputy editor of The Economist.
TLDR: (Generated with AI and edited for clarity)
With the importance of safety and security top of mind as AI systems evolve, Tom Standage of the Economist and Oren Etzioni of TrueMedia.org, had a discussion during the 2024 IA Summit about the increasing risks posed by deepfakes, particularly in political contexts. Oren explained how his experience in a high-level meeting on AI and disinformation with President Biden sparked the creation of TrueMedia.org, a nonprofit aiming to provide tools to detect deepfakes for the public. The conversation highlighted the potential harm of deepfakes in elections, the limitations of current detection technologies, and the need for real-time solutions to combat disinformation.
- Deepfakes Pose a Significant Risk to Elections: Oren emphasized that deepfakes have already influenced election outcomes, such as in Slovakia, and could play a pivotal role in future elections, especially in swing states or during critical moments like the time between election and inauguration.
- Tools Are Available but Not Perfect: While TrueMedia.org offers a free tool with state-of-the-art capabilities for detecting deepfakes, no detection system is 100% accurate. TrueMedia.org uses an ensemble model from major commercial sources and academia and follows up with human analysts to improve results.
- Preparedness is Crucial for Future Attacks: Although widespread disinformation through deepfakes hasn’t fully materialized, Oren warned that organizations and governments need to be ready for increasingly sophisticated and targeted attacks as AI technology advances.
You can see the session recording here.
This transcript was automatically generated.
Tom: I’m Tom Standage, and I work at the Economist — I’ve been writing a lot about AI lately, you’d be surprised to hear. My original plan was actually to work in AI, and I went to Oxford to study computer science in the late eighties when, of course, AI didn’t work. So that scotched that plan and I had to pivot to journalism instead.
But what’s been really interesting is in the last 15 years, going back and seeing what changed so that it did work. So I’ve been doing a lot of coverage of AI certainly in the last 20 years, but even more in the last five or so. Anyway, so I’m thrilled to be able to ask Oren Etzioni about his latest venture. I think you’re best known… Well, obviously, you’ve been associated with many organizations over the years. This was a bit of a swerve, I thought, because I’d spoken to you in the past when you were AI2, and then suddenly, you popped up with TrueMedia.org, which provides tools to detect Deepfakes. So, was there a particular event that led you to say, “I need to do this.”?
Oren: Well, Tom, it’s very simple. It’s the opposite of the trajectory you described. You see, I started working in AI for many years and I got a hankering to get into journalism.
Tom: Wow. It’s not a great business to be in, let me tell you.
Oren: I’m joking. So to answer your question more seriously, in the summer of 2023, I was in a meeting with President Biden, Governor Newsom, some key members of his staff and some AI luminaries, and I came in there because I am an AI optimist, I see so many possibilities, startups, public markets, what have you.
I came in there pitching an exciting idea. I came out of that very scared about deepfakes, and I thought, “Okay, deepfakes can have major consequences.” There’s a Cold War saying that words properly deployed are more powerful than bombs. And then, if you add to that, a picture is worth a thousand words, and I like to add that a video is worth a thousand pictures. Pretty soon there’s a huge risk there.
I came out of that meeting and said, “Okay, let’s survey the tools available to the general public, available to media organizations to access this.” And it’s a desert out there. There are tools available to three letter agencies, to major corporations, but they’re not tools available to election officials to keep people… to ensure the election is made. So as a result of that, I launched our nonprofit, and the origin story has to do with President Biden, but we are a nonpartisan organization. I don’t think either side of the aisle wants to win the election based on fakes, based on duping the public.
Tom: Okay. So that’s where it comes from. I think there are other tools, sets of tools out there. I DARPA has one, but you basically have to download and compile them yourself. So you wanted to just make this something that absolutely anyone could use. Is that the case now? Can anyone — because I think there was a point where you had to register to use the tool set — can anyone go to it now and just use it freely, or is it restricted in some way?
Oren: You’re absolutely right. There’s a ton of models. There was a DARPA program. There are things sitting on Hugging Face and so on, but if you’re a non-technical user, your choices are very, very limited. There’s some ancillary academics projects, then there’s us. We’re a free tool. You go to TrueMedia.org, in beta you had to have an account, now you don’t. Go from your phone right now to TrueMedia.org, and you can see that you can either upload a social media URL, analyze it as fake or sign up and then upload a file, a video, audio image, and decide. So we have tried to make this as frictionless as possible.
Tom: Brilliant, brilliant. Before we go any further, this countdown clock here says 5:54, which is not the time, and it’s not counting down. So please could you make it count down to wherever you want me to stop? Thank you.
So, just quickly, before we get into the technical stuff, where’s the funding coming from for this?
Oren: This was founded by Garrett Camp, who was the co-founder of Uber, and as a nonprofit is funded by his-
Tom: So he’s like the Waz of Uber. Okay, fair enough.
Oren: Well said, well said. He’s been a huge inspiration and he’s funded it as a philanthropic effort.
Tom: Right. Brilliant. Okay, so how well does it actually work? Because there’s a lot of people saying, “Well, teachers want things so they can see where their essays have been written at school by using ChatGPT.” And we know those tools basically don’t work. So do these tools work? Can you actually be confident when you put something into this and it tells you it is or it’s not a deepfake, are you confident that it’s giving you the right answer?
Oren: Do the tools work? Let me give you a simple answer, yes and no. Now let me explain. Yes, in the sense that we have state-of-the-art capabilities. We have an ensemble model that hits major commercial vendors. Some of them are actually in this room as well as models from academia. We collect the results, form an analysis, and we can see that we have state-of-the-art capabilities well above 90% correct on data from the wild. That’s the good news. That’s the yes.
The no part is that people really want certainty. They want to have the right answer, and often we can’t be 100% sure, we mitigate that with our user interface. So for example, when you give a query, sometimes it’ll be red, we’re quite confident it’s fake. Sometimes it’ll be yellow, we just can’t be sure. So the reality is that AI tools of the present in the future, like ChatGPT don’t always give you the right answer. You have to do some additional work. And just one last point. So as a result, we have now a human analyst team that does a fast follow on any query to analyze and make sure that we gave the right answer.
Tom: Brilliant. Now, I believe you also put… when I’ve used it, you get, “Here are five tests and four of them think it’s a fake one of them doesn’t, and here are the confidence levels in each one.” So when I was talking to the guy at DARPA about this, he was saying that individual tools may not be that good, but if you have a suite of tools, you can get much higher overall accuracy. Is that part of your model too?
Oren: That’s exactly right. So we try to be very transparent and give you the details underneath. What we see is on data queries that come in from leading media organizations, from the general public, things that are happening in real time and social media. We see that the ensemble, the suite like you said, is substantially more accurate than any individual solution. And that puts us in a great point. The other thing that happens, which DARPA did not have access to is as we get queries and the number of queries is increasing very rapidly, we are able to train and fine-tune on those so the models are rapidly getting better.
Tom: And presumably, if you see something, there’ll be a flood of people sending the same image or the same clip and saying, “Is this fake?” Can you just recycle the old answer? You’re not going to run the query again every time, are you?
Oren: Of course not. So we do what’s called caching, which cuts our costs and also improves the experience for the user. So if you come in and do an answer, usually it takes a minute or so. If you get the answer instantaneously, you know it’s been cached, somebody else has queried this before.
Tom: Great. And what’s takeup been like you said that news organizations are using it more. Is it what you expected? Is it enough? What is enough?
Oren: I’m very pleased with the uptake from NGOs, from news organizations and from individuals. We have thousands of organizations using it, News Guard, Media Literacy, a whole range of… we participated in takedowns of fake news networks in Europe in the Indian election. I’m very pleased with that.
What I’m not pleased about is that I’m really worried we’re 34 days in, I’m worried about what’s going to happen in the 48 hours before the election and in the two to three weeks afterwards or the time between election and inauguration. There’s a potential for unprecedented disinformation, and this isn’t… people who know me, I’m not an alarmist, the sky is falling. Let me just mention two key examples because in our previous conversation-
Tom: Is it going to be Slovakia and Taiwan?
Oren: Slovakia is one. So let me just talk about that one. And that’s in 2023. In the quiet period before the election, they have a 48-hour quiet period, an audio recording was leaked. One of the candidates talking about rigging the elections, it turned out to be fake, but for a variety of reasons, he lost the election. So a pro-Europe candidate lost to a pro-Russian candidate. And this is arguably one of the key factors. Very clear example of very effective election interference, whether it fully drove the outcome or not.
Fast-forward to January 2024 in this country, in the New Hampshire primary, as you know, there was a robocall to tens of thousands of people of President Biden, I’ve listened to it, probably you have as well. It sounded just like him. He said malarkey and all that stuff in his inimitable way. Again, there are no stats on the exact impact of that, but I’m very worried about a election interference targeted to specific swing states, targeted to specific polling stations.
And so what I’m not happy about is the degree of preparedness that there is for very specific targeted attacks. They may not materialize, but I think it behooves us to be ready.
Tom: Right. I was going to ask. There was a lot of concern, there has been a lot of concern. This is the biggest year for elections in human history. More than half the population lives in a country holding elections, although some of those should be called “Elections” like Russia, who knew who was going to win that one? And Rwanda, 99.7% voted for the incumbent, et cetera.
But at least voting of some kind is happening in countries where more than half the world’s population lives. This is the first time in history that’s ever happened. And one of the things that people started saying is, “Oh no, this is happening just when AI is there and can produce all this misinformation.” But there are people who argue that actually this is a kind of dog that hasn’t barked. You’ve got that example in Slovakia.
There was some stuff happening in the Taiwanese election in January, which there was Chinese misinformation, but it was quite bad. And so you can make the optimistic case that actually this hasn’t been the terrible disaster that we thought where people were having their minds changed, those pictures of Donald Trump walking around with cats under his arms, you’ve probably seen those. They have a kind of sheen of generative AI to them. No one is fooled by this. So what do you say to people who say, “Actually this turns out to have been less of a problem than we thought?” It’s still too early.
Oren: I think there are two points to make about that. I think that it’s actually a fair point that a tsunami of these videos and audios that are so easy to generate has not emerged. So it’s not the case that we see millions of them, even though they could be generated for just a few dollars or even for free. We see millions of these per day.
However, I think that that’s the wrong metric. Another metric is how many eyeballs are on something when Elon Musk, that bastion of responsible journalistic integrity, free speech, shares a fake video of Kamala Harris without noting that it’s a parody and it’s viewed tens of millions of times. Now, were people confused by this? Did they think it was really her? I’m sure some were, maybe not as many, but did this affect her brand? Because at the end of the day, so much of the election process is a kind of branding exercise.
So there’s a lot of stuff seen that affects people’s brands. And so some of the things that you see as kind of silly or ironic, Vice President Harris with communist symbols or former President Trump with teenage girls on the Epstein plane, even though you find out quickly enough or you can see that they’re fake, to what extent does it affect their brand? So I think that overall it is a major issue, not as major as somebody being assassinated, but still very important.
There’s another point that I want to highlight, and it was in the news literally today. Our focus at TrueMedia.org is on political deepfakes, particularly this upcoming election and this year. But there are so many other cases. So Melissa Hutchins is here working on how it affects women and girls. There was a startup recently funded how it affects celebrities.
Today, there was news about simulated kidnappings, which is a growth industry. You get called, you can see video of your loved one, and it’s fake and it’s all too easy to manufacture. So this problem that we are at the forefront of is not going away and it’s not getting better. It’s getting worse. So I won’t argue on is, “It as bad as people thought it was?”.
Tom: Right. It’s still a problem.
Oren: What’s clear is I like to say, don’t bite my finger, look at where I’m pointing.
Tom: Okay. Well, another thing that’s said about this is it’s very clear that certain people in this country are very resistant to fact-checking. We saw this last night where somebody dared to point out that something that JD Vance said in the debate was fact-checked and fact-checked in real time. And he literally says to them, “You said you weren’t going to fact-check me.” And they cut his mic.
Renée DiResta said to me when I was talking to her about this, she said, “Some people are highly resistant to fact checks if they don’t like the fact-checker.” So what can you do about that or is that not your problem? Basically how do you get the people who need to be told that things are fake to hear that it’s fake? And then when they hear that it’s fake, say, “Yeah, but some democrats probably saying that.”. How do you get through that wall?
Oren: There’s a broad set of problems here, you’re absolutely right. And it says right on top of our homepage, we’re nonpartisan, but hey, we’re from Seattle.
Tom: But the facts have liberal bias famously, right?
Oren: Yes.
Tom: Those of us in the reality-based community tend to be on one side of the aisle.
Oren: You’re absolutely right that there’s room for suspicion. Here’s what we’ve done and it’s not a perfect solution, but it’s our approach. We’ve focused on identifying a technical observation. You can think of it as metadata to associate with your video, audio or image, “Was this substantially manipulated? Is this completely fake? Was the audio fake? Was the video fake? Was the image changed?” Et cetera. That is a technical distinction. And then you can decide how to interpret that in a political context or another context. So we don’t even do fact-checking because sometimes the facts can be very clear sometimes it can be misleading, it’s a slippery slope.
Tom: That’s it. You’re providing the raw material for other people to say that.
Oren: Exactly. As a technologist, we are focused on high integrity, high transparency analysis of a technical distinction. Was this faked by AI?
Tom: Right. So it sounds like fact-checking is not your department. I’m reminded of a quote from the 2016 election from Clay Shirky who said that the Democrats had taken a fact-checker to a gunfight, which is a similar kind of problem, but I think 2016 is informative in another way too, which is that the extent to which there was strange stuff going on social media and sort of Macedonian teenagers or however, whichever version of that story you find sort of most compelling, that only actually emerged in 2017.
So is it possible that we actually can’t see when we’re in the thick of a campaign, what is happening and what influence it might be having? And we might not find out actually until next year the extent to which deepfakes were or were not circulating and were or were not changing people’s minds, is that something that you worry about?
Oren: Absolutely. So what’s become clear to me is I’ve delved into this space and there are organizations like the Microsoft Threat Analysis Center, is that there are influence operations that have been documented and uncovered and they work on different time scales with different targets and goals. For example, just to sow distrust, to undermine faith in institutions in our society, which I very much think would be a bipartisan issue. We don’t want that.
And so the approach that we’ve taken is to make sure that the analysis we do is available in real time, not three days later, not if you submit it to some organization that then takes a few days and publishes a result, but you get that technical assessment in real time. We think that’s essential. The other point that you made also think is really important, we’re all subject to what’s called availability bias. I think, “Okay, so what are the deepfakes that I’ve seen?”
I’ve seen many more deepfakes than any of you because of Delta. There’s a possible exception of Molly who’s our head of product here in the audience. And I can tell you that I’ve seen so many that have been viewed millions of times that you have not seen because they were on Telegram, because they’re relevant to the Middle East or the Ukraine, because they’re in Russian, because they’re in Persian, because they’re images that are absurd to us, but would be convincing to other populations.
So we already know that taken collectively, these media items have been viewed billions of times by people who aren’t us. I dare to say that most people in this room, maybe everybody in this room are not the targets. And as a result, the influence operations, which can be highly targeted are not targeted to them. But please don’t make the mistake that because you don’t see a lot of deepfakes they aren’t out there.
Tom: There are other people seeing it, right? Another weird thing about this field and about 2016 in particular, is there still no scholarly consensus on whether funny business on social media made a difference to the outcome or not, which is astonishing. But the literature is literally, there’s a bunch of people who say, “Of course it made a difference.” And there’s a bunch of other people who said, “No, it didn’t make any difference.”
So we could also find ourselves post this election in a world where there are lots of people who say that deep fakes swayed the outcome, and there are a lot of other people who don’t. And we may never agree on that. Do you think…? It’s just very, very weird that there’s no consensus on whether this makes a difference or not? Do you think it actually does?
Oren: I think that quantitative social science is hard to do and even harder to believe in after it’s done. Both my parents are sociologists and political scientists and love them dearly, but I grew up with a healthy skepticism of that. What is clear to me is that we’re on a technology exponential that we’ve been hearing about all day where these items are getting even, in the last few months, increasingly credible, increasingly powerful, increasingly easy to manufacture. That’s point one.
And point two is that we have a chance, it’s very hard for me to estimate, is that a 1 in 1000, is it a 5% chance? I don’t know, to have election interference, to have market interference? So there was the photo of the Pentagon, just one photo of the Pentagon being bombed, resulting in a drop in the markets. I think it was 2021. So what is 100% clear to me is if the shoe hasn’t dropped, to use that phrase, yet, it’s coming.
And the examples, Slovakia, Taiwan, as you mentioned, Pakistan had an interesting twist on this. So what happened in New Hampshire? These are the political ones. Sooner or later we’re going to have a major impact and we just need to be ready without violating free speech.
Tom: And which countries do you think are doing a good job of this, of fighting disinformation? I think a lot of people point to Taiwan, they used to point to Brazil, but I think the judge there may have sort of overplayed his hand a bit, but who do you think the US and other Western liberal democracies could learn from in this regard?
Oren: That’s the easiest question you’ve asked me so far. Nobody’s doing a good job at this, they’re doing a terrible job.
Tom: Not even Taiwan.
Oren: No, because we’ve seen so much during the Taiwanese election that was viewed many, many times. Again, people are making efforts and I think the reason is that we need the key social media networks to step up. And again, I’m not suggesting that they censor stuff, I’m suggesting that they serve appropriate metadata, appropriate information to it. And to some extent, YouTube has started doing this.
Tom: And Community Notes, Twitter is terrible on this in a lot of ways. And has pulled out of the EU program to stop disinformation and all the rest of it. And obviously Musk himself is spreading lots of crazy stuff. But Community Notes is actually pretty cool, isn’t it? It does mean that stuff gets labeled and you can tell that it’s bogus.
Oren: I think Community Notes is helpful. We actually download that information on an almost daily basis. I think it’s also helpful for Freelon Musk and Twitter.
Tom: It’s window dressing.
Oren: It doesn’t impact their bottom line. Exactly. My old friend, Ali Farhadi earlier talked about open washing. There’s deepfake washing.
Tom: If you try and report something on Twitter there are lots of categories you can report it under, but it’s fake. It’s not one of them. So misinformation is literally not one of the categories. You cannot report something as disinformation. So that kind of tells you something.
Just generally, we’ve got three minutes go. Where do you see all this going in the long run? I think if you look at the history of media, the 2000-year history of media, most stuff that you heard was not true, it was gossip and all rest of it. Most stuff that was printed… the printing presses was not true for a long time. There were lots of stupid pamphlets floating around about strange births and mysterious beasts and a lot of it was nonsense.
And it’s only really when you got monopoly media, large newspapers, TV and radio where there were very few of them and they had a self-interest to be right more than they were wrong. So that was, I think in a way, an anomalous period of history that between about 1850 and about 2000 and during that period, and it’s a period where many people, including me, grew up. You could basically trust what was in the newspapers and on the TV and on the radio, by and large it would be right.
So I wonder whether we went from a world where the default was you can’t trust it, to the default you could trust it, briefly. And now we’re back in a world of you can’t trust it and it’s very hard for us who grew up in that period to unlearn it. What do you think of my historical thesis?
Oren: I think it’s actually very insightful in the sense that those who are unaware of history are doomed to repeat it. At the same time, I think that there’s always something different. So what’s different now? We’re visual creatures, we’re sensory creatures. When you see something, you can’t help but respond to it. And particularly if you’re subject like we all are a confirmation bias if you already believe in the message.
So I think that the problem that we’re facing is going to be different because of the ability to do targeted attacks because of how much is happening over social media. And it’s not just politics. That’s my focus. But again, the first time that you receive a phone call from a loved one and it’s not you, the first time that you hear, “No, it’s not that your credit card was stolen, is that somebody got access to your bank account by pretending to be you.”
I went to my broker, I won’t name them, and they said, “Oh yeah, we have audio verification.” I said, “You’re kidding. Turn that off, for me.” And so, just like some families do, I have now a special code phrase that I use to identify myself when I talk to them, because don’t count out auto verification. So there’s a whole industry springing up like the cyberware, cybersecurity industry to deal with these issues.
And I want to end, at least my comments, by highlighting somebody said that disinformation is the malware of the 21st century. What we’re about to see, even if we go to a certain level of distrust, is potentially very problematic on many arenas in the legal system for women, in just phone calls that we receive in scams, and in politics. What I really hope is that over the next 34, 60 days, this won’t go down as an election that was tipped by deepfakes and that’s our mission at Trimedia.org.
Tom: Great. Oren Etzioni, thank you very much.
Oren: Thank you.
Related Insights
