Maybe it should resolve NO. The vast majority of comments on this video appear totally unaware that it's AI, therefore NO, people's first thought is not "is this real or AI".

https://www.youtube.com/shorts/GmoQW683Ahk

I think this is actually proof AI video is already more than good enough to fool the average YouTube user. Just look at the comments.

Clearly the folks who commented on this video (and on the many other videos from that channel) should be asking themselves "is this real or is it AI".

*Edited for clarity

@SIMOROBO Ok, I'll bite.

AI is amazing at generating images. I failed to get more than 50% of the answers correct on the AI Art Turing Test on Astral Codex Ten.

https://www.astralcodexten.com/p/ai-art-turing-test

AI currently sucks at video generation. It can generate gorgeous images and then make a short clip with slight movements of the image, creating a video that feels very static. Longer videos generally have 0.5-5 second segments with jump-cuts between them. Sure, there are human-created videos like that, and those could be confused. And to be fair, the video Yudkowsky originally retweeted has this flavor, but contains still a lot more complex movements than any AI video I've ever seen.

Can you imagine a version of the AI Art Turing Test but for categorizing videos (not necessarily art)? I can already imagine it:

• Video 1: A complicated choreographed dance with complex movements of a group of dancers

• Video 2: A video of a beautiful bird that is mostly static

• Video 3: A clip of a football (soccer) match with an impressive goal, showing multiple players on the field

• Video 4: A video of a flower blooming with a blurred background

• Video 5: An impressive aerial arc shot of a beautiful mountain without cuts

• Video 6: A woman walking, and turning around. When turning, the back of her head transforms into her face.

I think I can pick the AI-generated videos from that list. And sure, maybe I'll miscategorize a human-created video with a lot of static shots as AI generated.

I think Yudkowsky's take is surprisingly good, and maybe his prediction will come true in a few years. We're just not there yet.

@FlorisvanDoorn OK, I'll bite back ;)

The obvious problem is that there is a lot of room for interpretation in this market. Here are the nine combinations that I think rational people could have expected was the true spirit of this market. Open to hear about other interpretations.

AI video generation is at a level of quality that makes it impossible for [anyone, most people, some people] to be able to tell what is real and what is AI for [every, most, some] subject[s].

Personally, I think we've hit the "most people, some subjects" mark based on that YouTube channel alone. I would certainly feel more confident about the resolution of this market if we had made it to "most people, most subjects". In all honesty though, I thought the current state was precisely the minimum required for a YES resolution. If most people are sometimes fooled, they should ask themselves if it's real or AI.

But I understand if some or even most participants here interpreted it at a higher standard of that definition. I just hope we're not moving the goalpost which appears to be overly common with AI and Turing-like tests.

Meanwhile, this makes me wonder how people here would think such a market should resolve if it was about image generation instead. And since you mention it, I would be curious how many here would have changed side after taking the ACX Art Turing Test. Maybe it's not new technology but a reality-check demonstration that is needed to correct one's biases.

I think with image generation we are simultaneously at "anyone, some" and "most, most", but perhaps even just "anyone, most".

@SIMOROBO

I think you're right that there is a (very) narrow slice of videos can fool a lot of people (maybe most, as you said). However, I think this is a very small slice of "beautiful videos". I'm pretty confident AI is not currently able to create a decent alternative for 95+% of the (top) videos on nature subreddits like r/natureisbeautiful or r/natureisfuckinglit

Why I think this is clearly a NO:

Go to /r/natureisfuckinglit and sample the comments of some of the recent posts that show any of the coherency in this moth video. Note how many posts have upvoted comments that mention AI or the possibility of the video being fake.

By my count, I've looked through a few dozen posts and found 0 mentions of AI or the possibilty of the gif being faked. AI video is simply not there yet for people to be extremely skeptical of all video content, which is what this tweet is predicting.

The question boils down to whether "Eliezer was importantly wrong about something, e.g., AI-generated video still sucks, or still couldn't be the cause for serious doubt about whether some random moth footage was made with a camera or not."

The answer is obvious that he wasn't importantly wrong about anything. There have been numerous instances of videos posted that are AI-generated but indistinguishable from reality and even more instances of videos posted and people arguing that they are/aren't AI.

Eliezer was correct with this tweet.

@RiskComplex The question isn't about AI videos in general, it's about every video which is as "beautiful" as the moth video being up for debate whether it's AI or not. What is "beautiful" can be very subjective, but for the question to resolve YES, then AI should at least be able to generate flying moths that look something like the video. But they can't, so this question should resolve NO.

Also, you saying that Eliezer was correct doesn't make sense, because resolving this YES now would mean that Eliezer was wrong, as he predicted AI videos would become that good 2025-2027, not before.

@RiskComplex Yudkowsky made a specific prediction about what my "first thought" would be "anytime" I saw a video "this beautiful". Not my second thought. Not for only some videos.

If he wanted to make a different prediction, that AI generated video would not suck, he could have used different words.

There are many people making vague or rhetorical prophecies and retrofitting them to reality. It's a silly game. We should hold Yudkowsky to higher standards than astrologers.

@MartinRandall Yudkowsky isn't the market-maker. And the market maker states that they will decide if this was a 'good take': YES if AI video is that good and NO if he missed something critical ("AI-generated video still sucks, or still couldn't be the cause for serious doubt about whether some random moth footage was made with a camera or not").

Those are the criteria for YES and NO, whether you like it or not.

@Pazzaz The market maker implies that it is indeed about AI videos in general. "AI-generated video really is that good" and "e.g., AI-generated video still sucks, or still couldn't be the cause for serious doubt about whether some random moth footage was made with a camera or not."

This is referring to the quality of AI video in general.

@RiskComplex "is that good" = as good as the moth video. It specifically calls out "moth footage". It's about videos which are similar to the moth video.

@RiskComplex I was mostly replying to your comment that "Eliezer was correct".

I think it's ambiguous now, but somewhat close to YES. There were cases of generated video being treated as real on reddit with thousands of upvotes.

e.g https://www.reddit.com/r/singularity/comments/1gh5p1y/ai_generated_video_gets_thousands_of_upvotes_on/

some cases are still ambiguous imo, like this one https://www.reddit.com/r/Damnthatsinteresting/comments/1fbmtiu/man_giving_water_to_a_snake/

weirdly static scene, bokeh balls not flashing in and out, but good furniture, but bottle that crumples. Like, i honestly don't know

and also some people suspect real videos now e.g.

https://www.reddit.com/r/ChatGPT/comments/1g3cjnr/is_this_video_ai/

I'd resolve this as YES, but if you count more uhh more common paranoia, then eh. Not yet, it's not really common

@Lavander https://viralhog.com/watch/file/395937723 same guy giving another snake water

ViralHog - Watch

@Lavander I reviewed the polar bear video and my first thought was "huh, is that what the Arctic looks like?". So not what Yudkowsky predicted.

Here's my pitch for why this ought to resolve YES:

The tweet does not say "AI video will be just as good as this moth video". It says "your first thought will be to wonder". Furthermore, the market uses even broader language: will the tweet "hold up"? Is it "a good take"? Could AI-generated video "be the cause for serious doubt"?

Every month or two a new AI video model is getting released. I keep seeing amazing new examples on my Twitter feed. When I see a random cool-looking video, my first thought is already to consider whether it's AI-generated or not. Sure, usually I can still tell one way or the other, after watching carefully. But this tweet was still a good take.

"hold up" is also super open to interpretation

@EMcNeill But the tweet also says that "anytime" you see a video like the flying moth video, you'll wonder if it's AI generated. But generative AI can't even generate flying birds, let alone flying moths. There is no reason to believe that such a video is AI generated.

This should resolve NO (if nothing changes before 2025).

@EMcNeill The AI Videos that have proliferated in the wild are just awful. There's a missing middle of videos that nearly pass, which makes me confident that lifelike AI videos pretty much only exist as promos. There has been nothing in the ballpark of that Moth video and I have zero skepticism when encountering videos with that kind of engrained detail and coherency.

This reddit post is evidence in favor of a YES resolution IMO https://www.reddit.com/r/ChatGPT/s/lSCArBlw5d

@DylanSlagh There is a world of difference between skepticism for a distant out of focus shot of a bear moving and a coherent high detail shot of multiple moth species taking flight organically. AI is not there yet.

@GeorgeVii can you share your view on the market as of now? Does the video released today from Meta Movie Gen meet your criteria to resolve yes?

I would like to buy more No but I have a lot of uncertainty about your view on the state of the art.

@aaron how can a single (likely cherry picked and tweaked) demo resolve a claim about videos in general?

@AIBear This is implicitly a question about cherry picked videos, no? It's about "if a realistic looking video goes viral, was it likely made by an AI?". Someone who wants to make a viral video can generate a lot, for many prompts, tweak them a bit, etc before releasing, and I think it still meets the spirit of Yudkowsky's prediction

@NeelNanda I think you're using cherry picked in a different sense: the demo was all the videos Meta found most impressive, which would be different from generating many potentially viral videos for a given prompt and picking the best one

@dflz No, I'm using it in the same sense. If someone wants to make a viral video, they can try many prompts, and generate many videos for each prompt, and pick the best out of all of them. Or just post several, and eventually one will go viral.

You also get the selection effect that, even if no one viral video creator is doing this, if there's thousands of them, you'll only see the videos that go viral, which is an extreme form of cherry picking for impressive videos.

@NeelNanda I mean the demo itself could be cherry picked so that only videos that look good for the model

For instance, if a story telling robot could only write stories about cats, and the demo only showed cat stories, that would be cherry picking.

Compare to a model that could do a great story 1/3 of the time, and then cherry picking the great third of stories.

In this case, I think it's about cherry picking capabilities of the model, vs cherry picking an individual attempt

@dflz Also to add, if creating one good video requires $100k in cloud credits and human evaluating 800hours of videos, you likely don't need to worry that a typical viral video was produced this way. But I can totally imagine Meta spending an order of magnitude more to produce a demo.

@NeelNanda Another part of my concern is procedural. The model is not out, you currently don't have to worry abou AI generated viral videos, therefore the question shouldn't resolve at this time. Discussing whether the question should resolve YES if the model is released and performs this well (or whether at least somewhat public release is required) makes sense. Resolving now would however be premature.

There's a long history of AI companies faking their demos, so some caution is warranted.

@aaron I would be really surprised if the meta model resolved this positive, since it is about as good as Kling 1.5 which has been publicly accessible for ~1 month and has not caused a positive resolution.

@aaron I think there should be an AI-generated "insect flying video" or at least a "bird flying video" that is as good as the original video, and I haven't seen a model that does that. Having a model that can create beautiful mountains or fish is impressive, but it's not what the question is about.

@Pazzaz This what I get if I literally type "beautiful video of 15 different moth species flapping their wings, professional photography, 8k, trending on twitter" into Hailuo ai

@AIBear The cost point is fair, and I'm not arguing that this should be resolved YES now, I agree that's silly before the Meta model is publicly available and can be tested.

I am just arguing that the bar for "many viral videos are AI generated" is way lower than "this thing will reliably give you a fantastic video on a reasonable prompt in a handful of attempts", because the process of virality and commercial incentives give you a ton of selection power beyond an individual, not very motivated, user of a model

@NeelNanda none of the models publicly accessible could produce that moth video (without using some additional trick like first rendering it 3d and then using vid2vid). I strongly suspect Sora/Meta would do no better.

@AIBear There is no requirement that all videos 'in general' must be of this quality. Only whether AI can generate a video that is indistinguishable from a natural video by your average Twitter audience.

@RiskComplex @AIBear @dflz @aaron @NeelNanda @Pazzaz I questioned how this would be resolves months ago:

Is this market about whether this was a good take regarding 2024 (1) or 2026 (2) or whether Yud thinks he was "right" regarding 2024 (3) or 2026 (4) or whether @GeorgeVii thinks this was a good take regarding 2024 (5) or 2026 (6) or whether @GeorgeVii thinks Yud was "right" regarding 2024 (7) or 2026 (8) or whether @GeorgeVii thinks Yud thinks this was a good take regarding 2024 (9) or 2026 (10) or whether @GeorgeVii thinks Yud thinks he was "right" regarding 2024 (11) or 2026 (12) or whether the tweet is generally regarded as "right" in 2024 (13) or 2026 (14) or whether Yud or @GeorgeVii think the tweet is generally regarded as correct in 2024/2026 (15-18)?

Maybe noteworthy: There was no response, so we might need to take into account some mod resolving this according to how they interpret the question.