- A new Runway study asks volunteers to identify AI videos
- Clips were correctly identified only 57.1% of the time
- The videos were generated with the new Runway Gen-4.5 model
It seems we’re through the looking glass when it comes to the verisimilitude of AI-generated videos: in a new test run by AI video company Runway involving more than 1,000 participants, the volunteers were only able to correctly identify AI or real videos 57.1% of the time.
Considering that guessing at random would land you somewhere around the 50% mark, that’s a worryingly low figure. Even Runaway co-founder and Chief Technology Officer Anastasis Germanidis admitted to failing “quite a bit” in the task (via The Information).
If Runway’s own team members are struggling, then the rest of us don’t stand much of a chance. The fake and real clips were set to the same resolution and the same length, and the volunteers had 10 seconds to decide which were showing real people in the real world, and which were made by AI (reflecting the fast pace of social media scrolling).
The test was commissioned to mark the wider rollout of Runway’s new Gen-4.5 model, which the company promises offers “unprecedented visual fidelity and creative control”, as well as results that are “cinematic and highly realistic”.
‘Have a more critical mindset’
Watch On
Participants involved in the test fared best when human faces, hands, or actions where involved, with accuracy rates ranging from 58-65% for these clips. Germanidis says that might be because these videos are where ‘uncanny’ elements are most noticeable, even if the AI videos themselves are still of the same quality as real ones.
Germanidis is encouraging people to “have a more critical mindset” when weighing up anything they see online. “At this point we’re crossing the threshold, and it’s hard to tell apart generated from real videos,” he told The Information.
Runway is working on ways to reliably watermark AI output, with its videos already containing metadata identifying them as AI-generated by default. It’s something that governments are keen to see implemented, understandably, as the line between the real and the computer-generated disappears completely.
However, Runway Gen-4.5 isn’t perfect yet. Runway says it can still struggle with disappearing objects, causal reasoning (so doors might open before the handle is pulled), and success bias (so poorly aimed kicks still fly into the net in soccer scenes, for example).
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds. Make sure to click the Follow button!
And of course you can also follow TechRadar on TikTok for news, reviews, unboxings in video form, and get regular updates from us on WhatsApp too.
The best business laptops for all budgets
