Access to Gemini Ultra hasn't been publicly released as of December 16th 2023 yet but according to Google's 

 claims to beat out GPT-4 on various benchmarks.

It's a challenge to evaluate LLM-based chat assistants directly and their multiple methods, but one way was developed to use human preferences in a "Chatbot Arena" as presented in this 

 by the Large Model Systems Organization (LMSYS) team. There is a 

"Chatbot Arena Leaderboard" on HuggingFace

 from this team with this idea that uses human preferences to create an Elo rating to rank the different LLM-based chatbots.

As of December 16th, GPT-4 models sit at the top three spots with 

Fraction of Model A Wins for All Non-tied A vs. B Battles (2023-12-16)

Highest to lowest ranking of models goes from top to bottom

Bootstrap of MLE Elo Estimates (1000 Rounds of Random Sampling) (2023-12-16)

This resolves to "YES" if some version of Google's Gemini Ultra (if there are multiple versions) has a higher Elo rating than any OpenAI GPT-4 model on the public leaderboard at any point by April 30th 2024 (23:59 PDT). Note, this is only comparing 

 GPT-4 model and not necessarily the highest rank or most recent model.

This resolves "NO" if Google's Gemini Ultra appears on the leaderboard but does not ever score a higher Elo rating than an OpenAI GPT-4 model by April 30th 2024 (23:59 PDT).

This will resolve "N/A" if any of the following occurs:

There might be other edge cases that I might not have thought of but hopefully that covers any that are likely to occur. I may edit/clarify the description if there are new developments or suggestions by others.

The spirit of this is to have a public evaluation metric to compare Google's Gemini Ultra model versus one of the most capable models there is today (OpenAI's GPT-4 model).

The market below proposes the same question as the one listed above but resolves on after April 30th 2024 instead of the end of 2024 and only includes Gemini Ultra (not future Gemini models).

Similar to this market but asks about any model (from OpenAI, Google, etc.) that specifically beats the current (2023-12-20) version of GPT-4-Turbo: