Strange that OpenAI finished above 50%, seeing as that's the maximum! Probably a purchase of NO on another entry got it there?

@mods requires resolution

@Ziddletwix i resolved based on twitter buzz and search trends! deepmind and anthropic could've gone either way

@jacksonpolack can you share the exact data you resolved based on?

I feel like gigacasting should resolve this based on vibes

@AndrewG

It's about vibes, it's about feels, we stay biased, truth ain't real

Skip the facts, skip the proof, and vibe check's ours!

How about using this to resolve:

https://x.com/amir/status/1871259914091569608?t=verDkW_HAI2HADHhaR6Nlg&s=19

@SimoneRomeo where is anthropic

@Ellis not sure. I guess down in the picture as it's cut. But I don't have a subscription to the original article so I can't check

Is this largely about the performance of the largest parameter count models? Just curious how much of this will be determined by absolute performance of the largest models versus relative performance across size classes.

Based on twtr buzz, search trends, etc.

Heads up everyone that so far Midjourney, Suno AI, Elevenlabs, (briefly) Runway AI, Meta AI, and Microsoft AI all have search trends vastly higher than Deepmind and still significantly above Anthropic this year globally, the data doesn't lie

@TheAllMemeingEye appreciate the efforts to try to get data, but I think this is an incomplete picture. Midjourney, Suno AI, and some others are narrower companies with products more closely tied to their company name. The term "Claude AI" is above those you listed.

A product word associated with Deepmind (ie, "Gemini") dwarfs all of these.

However, this all challenging to draw conclusions from. For example the term "Apple Intelligence" is above all the terms except "Gemini", but rightfully no one is betting on Apple as in the top 3 pushing the frontier of AI research.

A top AI lab does more than just build one popular product. This means Google trends is one signal, but other signals that weight expert opinion or measure of research impact are also important. Unfortunately it hard to do this, and this market started pretty subjective. I would disagree that Stability AI belongs for 2022 as listed in the description.

My personal subjective opinion for 2024 is OpenAI and Deepmind seems pretty clear for top 3. Anthropic is possibly reasonable, though focus is narrower and their track record for AI innovation is shorter. I could see good arguments for Meta (just bought it from 3%->8%). In addition to their LLM work, they have a broad research mandate with longer history, and talented researchers in CV, robotics, and systems/infrastructure. (though have unclear long term vision for AGI and questionable general company practices).

It's less clear how things will look in future years. Also, it is worth noting that the probabilities of being in the top 3 is of 2024 is a poor reflection of faction of influence. Several organizations in single digits of this market are significant (along with the large set of orgs in "other").

@Dgfold thanks for the input :)

Personally I feel tying the resolution to as objective as possible criteria is important to avoid corruption and groupthink

I'll recheck Google Trends with terms like you suggested, adjusted to avoid non AI interference (e.g. zodiac sign interfering with Gemini)

Do you think the lmsys leaderboard and lead researcher h-indexes would be sufficient measures of the other factors you mention?

@TheAllMemeingEye I agree. But this might not be the market to do that as it is defined. One could make a new market defined around one or more specific metrics, but this one is vague and hard to bet on.

Just a note, Google trends will attempt to disambiguate the terms for you (eg, note how the Gemini version I selected is captioned with "Chatbot", not "Astrology"). I'm not sure how they do this, but I imagine it is more sophisticated than personal attempts. It is clear Gemini / Deepmind's products have large reach.

These are good ideas. Lmsys leaderboard is one metric that somewhat aligns with the public vibe of what AI is right now. But again, a top AI lab is doing more than just one product (in 2024 we aren't at AGI where anyone has one product that completely relegates everything else). Things like the research Meta is doing in computer vision, robotics, core product stuff like recommendations, or important infrastructure like PyTorch that every other org on the list uses all matter. Microsoft doesn't top Lmsys leaderboard, but their work is closely tied to the org that does (OpenAI), and in addition to user chatbots, Microsoft is doing cutting edge research in AI for sciences, AI for software engineering, and other areas. So lmsys is not a perfect metric that encompasses all of AI, but probably exerting a lot influence on a vibes-based market.

H-index is another estimate, or metrics like 2024 publications, etc. These all have their respective issues though if not all contextualized with other factors.

@Dgfold fair enough, you make good points

Going back to the actual market description, besides search trends it also mentions xitter buzz, are you aware of any way to measure this objectively? Is there a xitter keyword analytics site or something we could use?

@TheAllMemeingEye I think the X api is really closed off now, so hard to do that kind of site-wide analytics. Maybe someone with a decent budget could get some numbers on whatever you define buzz as. But you'll get different answers depending whether you focus on all of X (which will bias towards viral demos/marketing or controversy), or ways of filtering for different subcommunities, expertise, or topics.

It might be better just to focus on other/new markets defined along specific narrower metrics (for example the existing one on lmsys leaderboard, or following a mix based on specific google trends terms) rather than trying to grasp at one big quantification of the top 3 AI labs. I don't think you'll find good agreement on that question and it's not clear the goal.

@Dgfold "Gemini" could also be a search term used by people interested in astrology. I note, however, that "Gemini" ranks well above several others, despite no clear differences in the relative proportions of people assigned these star signs (https://genus.springeropen.com/articles/10.1186/s41118-020-00103-5).

@TomBurns thanks for investigating. Though as I noted above Google Trends will try to handle the polysemy for you if select from the dropdown when typing in a term.

long deepseek

@nosuch long qwen

I N/Aed the second deepseek bc duplicate

@mods as gigacasting is long off the site, may want to step in here before EOY to clarify if this market will be NA-ed or if a mod is going to handle the extremely subjective resolution

@Ziddletwix We'll just do a subjective resolution

@jacksonpolack I highly recommend instead using the reasonably objective criteria of search trends as mentioned in the market description (see my comment above)