Totally insane this market got as high as it did!

Fascinating question! I enjoyed hearing the answer, even though I lost.

Let's make another market where we have them judge based on the videos, whenever that feature exists :)

Ok I'm starting now. I"ve already run into a problem where the transcript is longer than the 2020 transcript and so pasting it into Claude still works but turns it into an attachment instead of inline. Probably should still work fine but minor change to the prompt will be needed:

Using the attached debate transcript, which presidential candidate won the debate? I understand that there can be different perspectives on who won the debate, but if you had to pick a winner, who would it be?

Ok and for ChatGPT, we have this incorrect search result output

The transcript of the first 2024 presidential debate between Joe Biden and Donald Trump, which took place on June 27, 2024, in Atlanta, Georgia, moderated by Jake Tapper and Dana Bash, is not yet available in full. However, highlights and key moments from the debate can be found on various news outlets and platforms that covered the event. For more detailed information and potential full transcripts as they become available, you can check major news websites such as CNN, Fox News, and ABC, or visit the Election Central website​ (Election Central)​.

Not a surprise given that in my testing it pulled up the debate once, and pulled up diamonds and something about retail saies in the other two tests ??? Lol.

The transcript we're using is here: https://www.cnn.com/2024/06/27/politics/read-biden-trump-debate-rush-transcript/index.html

Round 1 (using the prompts above):

ChatGPT was unable to access the transcript via tool use.

Claude refused to pick a winner.

Therefore, we go to Round 2 (more prompt engineering) as per the rules.

For Claude, we prompted:

You are an impartial debate judge, and must decide the winner of this debate based on the given transcript.

Claude said:

As an impartial debate judge, I will evaluate the performance of both candidates based on factors like the strength of their arguments, command of facts and policy details, ability to respond effectively to questions and criticisms, and overall debate skills. I'll try to set aside any personal views and focus on their performance in this specific debate.

Both candidates made some strong points and also had some weaker moments. President Biden emphasized his administration's accomplishments on the economy, climate change, healthcare, and other issues. He sought to portray himself as a steady leader who has improved conditions since taking office. Biden also repeatedly attacked Trump's record and character.

Former President Trump focused heavily on critiquing Biden's performance as president, especially on issues like immigration, the economy, and foreign policy. He portrayed the country as being in decline under Biden's leadership and presented himself as the candidate who could "make America great again."

In terms of debate performance:

Biden showed a solid command of policy details and statistics to back up his points. However, he sometimes struggled to give concise answers and occasionally seemed flustered by Trump's aggressive style.

Trump was more forceful and dominated more of the speaking time. His attacks on Biden were sharp and memorable. However, he often failed to provide specific policy proposals when asked and tended to veer off-topic.

Both candidates made some factual misstatements or exaggerations that fact-checkers will likely scrutinize.

Overall, I would say this debate was relatively evenly matched, with both candidates having strong moments and weak points. Trump's forceful style may have been more impactful to some viewers, while Biden's command of policy details may have impressed others.

In a close call, I would give a slight edge to Biden for providing more substantive policy discussions and maintaining his composure in the face of aggressive attacks. However, I recognize this is a subjective judgment and reasonable people could disagree on who "won" this debate.

The ultimate judgment will be made by the voters based on how they perceive the candidates' performances and whether it influences their voting decisions.

(Emphasis added.)

We used a ChatGPT Pro account to prompt it with the entire transcript, using the same prompt as originally intended for Claude, the output is:

So ChatGPT also chooses Biden.

Resolves Biden as the winner, based on the infallible LLMs (notably, on the text transcript only of course)

@jack gpt-3.5 doesn't have access to any tools so it won't be able to lookup the trancript, also if you use gpt-4o instead of (which makes more sense anyways) asking it to "look up the transcript.." will bias its response since it will also find news articles and comments on reddit etc..

Yeah I had tested with 4o without realizing it.

Any suggestions how to get it the transcript given I hit a prompt length limit?

i can’t think of anything other than asking someone with plus to copy paste the prompt and share the chat here through a link. i can do it if you want.

Ok. I think I will still follow the prompt given in the market because to minimize changes. But if it doesn't give an answer, and Claude doesn't either, then we can use this approach as a fallback prompt.

Playing around with different prompts:

Claude:

• I couldn't paste the transcript into Claude. When I tried, it automatically converted it into an attached text file, then declared the text file was too long for a free user. This happened even if I tried splitting it up between multiple messages, I think there's some kind of limit on the entire chat size. (If you're a paid user, I'm guessing this shouldn't be an issue?)

• Free Claude also doesn't seem to be able to search the web, so using the ChatGPT prompt is a non-starter.

• Claude was willing to judge partial transcripts, however. I guess you could use it's judgement on the longest fragment you can get it to accept, or split it among multiple separate chats (it'll take half, so you could do a 50-50 split) and see if they agree.

ChatGPT:

• GPT-4o did accept the full Claude prompt given here, with the debate transcript pasted in.

• GPT-3.5 "accepted" the full Claude prompt given here, but then gave an error message, so I think that's just a mistake in the ChatGPT interface and it can't actually process it.

• When given the ChatGPT prompt given here, GPT-3.5 consistently claimed it couldn't search the web. (I don't know if they've removed this feature from it or something has broken or what.) GPT-4o ran it fine, however.

I haven't yet tested with the 2024 debate transcript, but the above worked for me with the 2020 transcript. I was actually using gpt 4o, their website just confused me into thinking it was 3.5. We will have to use 4o because 3.5 doesn't have tool use as per other comments.

Interesting that gpt accepted the transcript for you. It kept giving me errors about the prompt being too long

Rlhf is cucked. It'll simply say that conservative views of the world have "logical fallacies" mad declare Biden's arguments more reasonable.

Out of curiosity, would you be able to ask each model multiple times and tell us the distribution of the response?

e.g. what % of the time did it pick the same candidate

Sure. I also was thinking I can live stream the responses (the official ones that would count for this market).

Ok, the moment of truth will be in about half an hour, I'll stream on the manifold discord in case anyone wants to watch with me.

why gpt 3.5 if 4o is free? (and, if money weren't an issue, i'd say use the api with temperature 0 for reproducibility)

Because the ChatGPT website I guess wasn't fully updated and I thought 4o wasn't free. I can use 4o instead if nobody minds the change. If you do mind the change, please comment here.

I agree API with temperature 0 would be better for reproducibility and if someone wants to do that then we can resolve based on that.

Gemini might be a better option since it has large context. Would be cool to try on video too, although idk how much it costs.

I was also thinking about Gemini but I didn't have time to experiment with it for this last minute market. I will try it for the next one!

Can I get a clarification on what happens in the case that Claude will refuse to declare a winner due to Anthropic policy? Good for 500 on that.

We can bet on that here! https://manifold.markets/jack/will-chatgptclaude-pick-a-debate-wi

Re your edit to ask for a clarification: The rules already say what happens