@NikitaBrancatisano Seems like the videos are generated by creating continuations of short videos, rather than being generated by text prompt

@TomShlomi You're right! I missed the text prompt bit 😔

Still, looks very promising

@ValeryCherepanov Weird, I'd have thought they'd prioritize this. I'm unclear how much I should trust this estimate and how much to update on if they are working on it right now

(@JoyVoid https://www.reddit.com/r/MachineLearning/comments/yw6s1i/comment/iwi1w7e/ Relevant)

@JaimeSevilla I might have gotten too excited.

The frame to frame consistency in the actual video is done through optical flow reconstruction, not stableAI like in the first clip shown

Phenaki shows very strong grasp of object perspective, rotation, zoom, layering, and permanency.

One can debate whether it is “crappy” (vastly less so than Dalle-1/mini) or “reasonable quality” (maybe) but it conditions on all prior frames and clearly passed the permanence tests.

@Gigacasting Hard disagree. It is pretty clear from its footage that it's just relying on the prompt to infer what should be there (see how the teddy bear never leaves water, but the scene just transition in a blurred fashion)

@JoyVoid This is why I was pushing for a pre-committed, specific test for object permanency. We're about to get bogged down into "what counts".

From an outside view, every debate about AI progress has the appearance of massive goalpost-moving (on both sides). Very frustrating.

I think the example from the market description (mountain being hidden by a cloud, then appearing after) should be agreeable? We have most of a year for someone to publish a model and someone else to try that prompt.

For examples of missing object permanency, the shape of the treeline noticeably changes when the rider crosses them in the top-center and lower-right images here.

A full occlusion of an object would be ideal. With a partial occlusion, it's hard to tell the difference between whether the model remembered the object (has object permanency) versus the model managed to infer a good enough filling from the parts of the model it sees (is really good at inpainting).

@FutureOwl I can't tell if I'm being pedantic and repetitive, or genuinely poking at an important issue. I apologize if it's the former.

As written in the market description, I think there's an important ambiguity. If the prompt is "a cloud passes in front of a mountain", then the test described, that "the mountain should still be there", is not in any way a test of object permanence. Of course the mountain is still there---the model can just read the prompt to see that there should be a mountain still there!

In short: it's very important that the thing that is permanent be something that isn't in the prompt, and is not inferable from the prompt+single previous frame.

@ScottLawrence As an aside (after watching those videos more closely), I also consider it an object permanence fail if the number of legs the horse has keeps changing!

But that's just beating a dead five-legged horse.

Mostly pedantic 🤔

The description asks for “most of the time”; it obviously passes that test in the videos and especially the longer video.

Picking the worst frame violates everything about the definitions (cherry picking edge cases)

Not opposed to calling this “crappy” but it’s missing the point entirely (or a massive difference in what the word “most” means 🤔) to focus on small objects or tree shape when he literally asked for “mountains” to not fully disappear “most of the time”

(No position as it’s very reasonable to go with either result.)

@Gigacasting I actually picked those trees because they were the only ones I could find where the horse went far enough that there was a full occlusion.

👑 anon paper

Vs. 🤮 meta hype

@Gigacasting Oh yeah I also just saw that paper on twitter, came here to see if someone had mentioned it.

Seems like Iclr is going to have lots of image2text stuff apparently.

Also after skimimg the meta paper while I think there's no clear distinction between gifs and videos , I do agree that only being able to basically specify the first frame of the video and have the video basically be interpolation from that is a big limitation and arguably doesn't fit the spirit of "arbitrary video" maybe.

But this anon paper doesn't seem to have that problem from looking at the abstract and examples so don't think whether that counts that will really matter for the resolution.

(not that I'm going to resolve yet but thinking about what my criterion will be).

These are not only NOT “arbitrary videos” they aren’t even videos at all.

They turn a prompt into an image, then do slight animation around it (similar to iPhone “Live” photos with a few bordering frames)

Very far from resolution—this is high-res gif animation and bears no resemblance to video

gif animations are literally videos lololol

It's still pretty bad with respect to the question, but this is Dalle1 level of crappy. Would be very surprised we don't reach stablediffusion level by June 2023 (I am less sure about an open-source/toxiccandy model by then)

@JoyVoid then why are you betting 500 on YES?

Apologies, I read that wrong XD

@JaimeSevilla Good demo, but still in crappy territory. Hoping to see a lot of progress in the near future. If you read the paper, you will see the limitations of this approach, mainly because we have GPUs with low memory. So this is not the path if we are hoping to get text-video with the current hardware.

@JaimeSevilla Yeah I saw that.

The demo looks nice but I'm not going to resolve yes yet.

Like it's possible that their model is good enough already but I'm fine whith waiting a few months untill we have better models and is more obious, especially if stability releases an open version I can play arround whith or something.

If we somehow don't get any better video models before the close date I'll think hard about it and ask people whether they think it should resolve based on this demo but that seems very unlikely.

@VictorLevoso In my opinion, this should not resolve yes yet, with respect to the "non-crappy" part of your question

@JoyVoid I think I mostly agree yeah and it does feel more like dalle-1 and looking at the paper seems it has some limitations arround how prompts work.

It is interesting that it seems to not fail obiously at object permanecy specifically but we also only have 4 potentially cherripicked 5 second videos so.

Btw to be clear I don't require an open source version to be released to resolve yes, it will just make my job easier and the resolution less contentious.

@ScottLawrence How much is the train or crowd of people allowed to change/shift? IE, if the train changes color or type of train, does it still count? If the number of people changes, or any of their faces/genders/clothing noticeably swaps, does it still count?

@LarsDoucet In what I described the train is in front---visible the entire video. If a visible object changes color or shape, that's a hard fail, right?

Since the point is to test object permanence, if the number of people changes when occluded by the train, that's also a fail. Similarly if it's obvious that a person changed. (For instance, if before the train passes there's a tall guy in a colorful hat, and that guy can't be found afterwords, that would be a fail from my perspective.)

Of course, the resolution is up to the market creator, I'm just proposing a test which I think reasonably probes whether the model "gets" object permanence.

I'm fine, by the way, with needing an extra prompt like "and no people vanish while the train is obstructing the view". However, needing to plug in object permanence manually (by saying "three people are visible before, and three people are visible after") shouldn't count.

Weaker version are possible. For instance, in a video of "bus passes in front of person", the person should have the same appearance before and after. Now there's much less information for the network to keep track of. Any network that fails that cannot reasonably be said to grasp object permanence, I think.

@ScottLawrence I like this test, just wanted to clarify. Because a lot of the video stuff I've been seeing tends to have a lot of shifting of these cases, but some of the newer models seem to be doing at least a bit better on that front. Nice and clear goal in any case.

@ScottLawrence Sounds like a good test to me.

Although I might still resolve yes if it can't do this but can do other similar things, or no if it can do the bus/train tasks but fails in similar tasks most of the time.

I think the "person still exists" level is enough, model has to show it can do basic object permanence, not be perfect at it, so let's say it's fine if people and objects shift a bit as long as they are recognizable as the same kind of person or object.

I'll be lenient whith things shifting but not whith things disappearing.

If it can do easy object permanency tasks whith 1 object but fails whith multiple objects I will likely resolve yes anyway as long as it's consistently good at the easy tasks.

@VictorLevoso Sounds fair.

One of the guiding principles I have is that it's absolutely essential that the model be able to perform object permanence for objects not explicitly mentioned in the prompt. So for instance, if I say "a bus passing in front of a woman wearing a red shirt", then I'm not impressed if before and after, there's a woman wearing a red shirt, since the model doesn't really have to "remember" anything, but can rather just refer back to the prompt.

Whereas, if I say "a bus passing in front of a woman", and the appearance of the woman is nontrivially the same before and after, that actually indicates to me that the model has captured the essence of object permanence. (Even if the shoes look a bit different before and after.)

That's the point of the "crowd of people"---it forces the model to invent a decent amount of information, and then get that information right before and after.

(I'm writing all this up less to convince/browbeat y'all, and more to record for me-in-a-year that this is the thing I doubt will be accomplished. That way I know when to be surprised/impressed.)

(While my NO bet is of course unconditional, I'll be properly surprised if this is done without any "hack", like first asking a language model to generate a detailed description of a scene, and then feeding that description into a video model. In other words, if a model manages to learn object permanence on its own.)

@M for the first thing I'll decide how to resolve based on the situation and how much info is available.
I'm unsure what to do in case of really short videos.
I actually expect that if we have good video models we'll have videos that are at least a bit longer(I mean transframer is already 30 seconds even if it doesn't seem to be great quality) and that even if its not open stability will release an open source version soon enough, so its likely not going to be a problem.
I guess I'll also ask people how they think the market should be reasonably resolved if its not obvious, and if its still unclear it will just resolve n/a or 50%.

@AlexWilson Lets say no, I guess. It has to at least be able to simultaneously do realistic and some other format like animation.