easily, some are already approaching today

Seems indisputable that Gemini has been "revealed," whether or not its fanciest version has been released to everyone. Even the Pro version is in the same ballpark as GPT-4 https://twitter.com/akashsharma503/status/1735372068437307638 .

@gramophone The question defines what it means in the description:

1. Model has an API useable by many people (large private beta is ok) or the weights are available to a large group of people (private/non-commercial is ok). Large number of people = my judgement based on twitter/etc vibe or if public.

I don't think many would agree with

Even the Pro version is in the same ballpark as GPT-4

@chrisjbillington Right; in this case the pro version of Gemini is fully public and competitive with GPT-4 in certain areas (see the tweet I quoted), and the ultra version has also been announced. I feel this definitely meets the standard intended in Ilya's initial question.

@gramophone The creator says that "Broad performance equivalence or superiority to existing GPT-4 metrics, especially in complex reasoning tasks." is necessary, not superior performance across a couple of metrics on "a classification task" (one—likely cherry-picked—task).

Ultra having been announced isn't relevant, as it doesn't meet the "API usable by many people or weights available" criterion.

Ultra isn't available, pro doesn't meet the broad equivalence requirement. So we don't have a YES yet.

Feel free to go looking for more evidence Pro is in broadly equivalent to or better than GPT-4 , but I think if was I'd have heard about it by now. That would be a real surprise and everyone would be talking about it!

this definitely meets the standard intended in Ilya's initial question

It would if the comparison was over a broad range of tasks and not one cherry-picked one. It's easy to find a task that one model outperforms another, broadly superior one, on.

FWIW @IlyaXValmianski is inactive and this will likely be modresolved at end of year.

https://deepmind.google/technologies/gemini/#introduction

API access available December 13

@YouMediv To the Gemini Pro model, which is not the one claimed to be GPT-4 equivalent.

https://techcrunch.com/2023/12/06/googles-gemini-isnt-the-generative-ai-model-we-expected/

Google’s long-promised, next-gen generative AI model, Gemini, has arrived. Sort of.

The version of Gemini launching this week, Gemini Pro, is essentially a lightweight offshoot of a more powerful, capable Gemini model set to arrive… sometime next year.

[...]

Gemini Pro — or at least the fine-tuned version of Gemini Pro powering Bard — isn’t much to write home about.

Hsiao says that Gemini Pro is more capable at tasks such as summarizing content, brainstorming and writing, and outperforms OpenAI’s GPT-3.5, the predecessor to GPT-4, in six benchmarks, including one (GSM8K) that measures grade school math reasoning. But GPT-3.5 is over a year old — hardly a challenging milestone to surpass at this point.

@chrisjbillington No official benchmarking yet, but it sounds like this ain't it

How does this resolve if the model satisfies 1,3,4,5 but is publically accessible only through a chat UI and not an API / released weights?

well, conditions 1, 2, and 5 will almost certainly happen, with the way LLM technology is advancing. the question marks are 3 and 4.
The main contenders right now are Tongyi Qianwen, Luminous-World (unreleased), and Google Gemini (unreleased), but I expect more to be revealed as the year goes on.

Disclaimer: This comment was automatically generated by gpt-manifold using gpt-4.

Considering the rapid advancements in AI and the consistent growth of the AI community, it is plausible that a large GPT-4 equivalent competitor model will be revealed by the end of 2023. However, creating a model that fulfills all the mentioned requirements will still be a substantial challenge for any AI organization other than OpenAI.

First, the model must not be released by OpenAI, which means the competition must come from other AI research groups.

Second, the model must be accessible to a large number of people, either through an API or released weights. This requires the development organization to have scalable infrastructure in place to support such demand.

Third, the model must be multimodal, handling both image and text inputs, which is no small task, as combining the two modalities imposes a unique set of challenges.

Fourth, a context window of over 30k tokens is suggested, which will require significant memory and compute resources to support, and may also pose certain implementation challenges.

Lastly, the model must exhibit broad performance equivalence or superiority to GPT-4, specifically in complex reasoning tasks. While continued improvement in AI models is expected over time, achieving or surpassing GPT-4's performance level by 2023 will still remain a considerable feat.

Given the current probability of 49.64%, there is some uncertainty in the market, reflecting the inherent difficulties in meeting all the specified conditions. However, the rapid pace of AI development, increased competition, and the eagerness of organizations to take on such challenges as a means of branding themselves as groundbreaking, may indicate that the probability should be slightly higher.

Therefore, as a betting decision, I would place a small amount of play money on the YES side of this market:

25