🏅 Top traders

My thoughts on some discussion below - deleting or rewriting an article without acknowledging past mistakes is not considered a correction, at least as far as my understanding is. Happy to be corrected on that.

It's worth noting the tweet still stands on twitter. I am not aware of any correction that was made. I believe from all perspectives, this market was correctly resolved - if over quickly.

However - in the future if someone tries this, I recommend leaving a buffer period for an outlet to issue a correction before resolving (something I should have done). Also consider removing NBC from the list of outlets to resolve from as they seem kind of clickbaity. Looking deeper into the policy of issuing corrections for each outlet might be a good idea as well.

twitter is a bit more permanent - https://twitter.com/NBCNews/status/1771386819919163712 Glad they didn't delete the tweet.

but just in case .. heh

@traders I am considering removing NBC from future markets as I agree their headline "U.S. government enters a partial government shutdown as Senate works to pass a deal" and article was poorly written. I was surprised and somewhat dissapointed by it.

I would love to hear some feedback on what folks think about NBC as a resolution source.

@houstonEuler You gave me 1 star. Can you explain your concerns about the resolution?

If those headlines are later updated to say something different should this be resolved differently?

@DanielBets It's an interesting point.

It shouldn't resolve differently, I don't think, as the idea is to minimize dispute here. We are betting on whether the headline will occur - not really the ground truth itself (which, imho, is often unknowable)

Unfortunately, articles change. I have an idea for automation to record it as soon as a link is reported.

@gpt_news_headlines that feels a bit against the spirit of the question, if an incorrect headline appears and then is quickly changed. It seems like the resolution should be evaluated at the originally stated closing deadline by whatever headlines still exist at the time.

@DanielBets Well, the question is literally "Will a Government Shutdown be Headlined by March 31st, 2024?" and that has indisputably occured.

The point of all my markets is whether the headlines will occur. I limit it to "Reuters, WSJ, AP, washingtonpost, NYT, BBC, ABC, NBC, CBS, and The Economist." to make sure it's not lightly chosen.

Do you think I should remove NBC from the list? I am considering it. I agree I felt they flaunted journalistic integrity with this headline.

@gpt_news_headlines I think the simplest solution is to leave a bit of a time buffer and evaluate the headlines a few hours after the initial event. That gives time for any initially sloppy/misleading headlines to be fixed while still holding the spirit of the event.

@DanielBets It's an idea worth considering, but the issue becomes that the new headlines may become time sensitive. For example, if a shutdown is over and it starts talking about it in the past tense.

Maybe I will emphasize at the top that we're betting that the headline occurs at any point during the market.

Trying this headline:

https://www.nbcnews.com/politics/congress/us-government-partial-government-shutdown-deadline-funding-midnight-rcna144746

U.S. government enters a partial government shutdown as Senate works to pass a deal

After a year of haggling and four stopgap bills, Congress finally struck a deal to fund the government — and failed to pass it in time. The shutdown is not expected to last long.

Congress failed to meet a midnight funding deadline.J. Scott Applewhite / AP file

March 22, 2024, 9:00 PM PDT

By Sahil Kapur, Frank Thorp V and Kate Santaliz

WASHINGTON — The federal government entered a partial shutdown early Saturday after Congress failed to pass legislation in time to keep a swath of departments and agencies open.

prompt:

..

Prediction Market Title:

"Will a Government Shutdown be Headlined by March 31st, 2024?"

Introduction:

Senators from both political parties are expressing concerns that disputes over the Department of Homeland Security's funding may lead to a government shutdown t\

his month, linked to conflicts in approving necessary spending bills by March 22. The warning comes after Congress narrowly averted a partial shutdown last week \

by passing a set of six mainly uncontroversial spending bills just before the deadline. This situation underlines a critical juncture where bipartisan disagreeme\

nts over immigration funding could trigger a shutdown, emphasizing the urgency and gravity of the current legislative deadlock.

Example Headlines for Market Resolution as YES:

"Congress Fails to Reach Agreement, Government Shutdown Begins"

"Department of Homeland Security Funding Stalemate Leads to Government Shutdown"

"Partisan Disputes Cause Government Shutdown, Services Halted"

"Partial Government Shutdown Official as Congress Misses Spending Bill Deadline"

"Immigration Funding Fight Results in Government Shutdown"

Example Edge Case Headlines for Market Resolution as NO:

"Congress Close to Shutdown but Passes Last-Minute Spending Bill"

"Government Shutdown Averted with Temporary Funding Agreement"

"Senators Reach Compromise to Keep Government Open Despite DHS Funding Debates"

"Partial Shutdown Threat Looms but Congress Secures Funding for Department of Homeland Security"

"Congress Nears Shutdown Deadline, but Key Spending Bills Passed to Maintain Operations"

Will the following headline and lead qualify:

U.S. government enters a partial government shutdown as Senate works to pass a deal

After a year of haggling and four stopgap bills, Congress finally struck a deal to fund the government — and failed to pass it in time. The shutdown is not expec\

ted to last long.

Storm clouds darken the skies above the Capitol

Congress failed to meet a midnight funding deadline.J. Scott Applewhite / AP file

March 22, 2024, 9:00 PM PDT

By Sahil Kapur, Frank Thorp V and Kate Santaliz

WASHINGTON — The federal government entered a partial shutdown early Saturday after Congress failed to pass legislation in time to keep a swath of departments an\

d agencies open.

...

import openai

import requests

import json

import os

import shutil

# Set up the OpenAI API

openai.api_key = '..'

def get_response(prompt):

response = openai.ChatCompletion.create(

model="gpt-4",

messages=[

{"role": "user", "content": prompt},

],

seed=0,

max_tokens=2048,

temperature=0,

)

return response.choices[0]['message']['content']

..
>>> qoai.get_response(open("tst.txt").read())

'Yes, this headline and lead would qualify for a market resolution as YES. The headline clearly states that the U.S. government has entered a partial shutdown, a\

nd the lead confirms that Congress failed to pass necessary legislation in time, leading to the shutdown. This aligns with the prediction market title asking if \

a government shutdown will be headlined by March 31st, 2024.'

>>> qoai.get_response(open("tst.txt").read())

'Yes, this headline and lead would qualify for a market resolution as YES. The headline clearly states that the U.S. government has entered a partial shutdown, a\

nd the lead confirms that Congress failed to pass necessary legislation in time, leading to the shutdown. This aligns with the prediction market title asking if \

a government shutdown will be headlined by March 31st, 2024.'

>>> qoai.get_response(open("tst.txt").read())

'Yes, this headline and lead would qualify for a market resolution as YES. The headline clearly states that the U.S. government has entered a partial shutdown, a\

nd the lead confirms that Congress failed to pass necessary legislation in time, leading to the shutdown. This aligns with the prediction market title asking if \

a government shutdown will be headlined by March 31st, 2024.'

>>>

@gpt_news_headlines This article has already been modified to have a different headline "Senate begins late-night votes on $1.2 trillion government funding bill". I'm curious if it would still pass now.

Can we please not do this? Having AI resolve markets based on headlines is ridiculous. You're correcting for the possibility of a humanly-biased, and therefore wrong, resolution by replacing it with a machine with a much higher probability of biased and wrong resolutions, but because the biases are arbitrary machine learning biases instead of human ones they don't count?

@jacksonpolack If an AI had a misresolution rate comparable to a human, would you be ok with it? I'm not involved with this bot, but I find the promise of it fairly compelling for potentially generating more high-quality markets. Like, it's not trivial to create and run a market on Manifold. There are many markets that I have chosen not to create due to maintenance and technicalities that I think the best version of an AI arbiter could have handled.

This market absolutely highlights the pitfalls of an AI-run market, especially one that only triggers a resolution off of a single headline, but I do think there's room on Manifold for experiments like this, especially one as fully-disclosed as this market was.

@Charlie I'd say that as long as the precedent is clear as to whether common sense will override the criteria in various circumstances (e.g. incorrect headlines later retracted), which media sources count, etc, it's fine, and I kind of like the experiment and am interested to see how it pans out.

The market title doesn't ask if things will happen, it asks about whether they will "headline" which is then defined in terms of how ChatGPT will interpret headlines. In this case a false headline was printed, so even if ChatGPT wasn't involved a YES resolution seemed likely.

Honestly it seems like a decent way to make resolution more objective, at the expense of weakening the strength with which the market is a proxy for what we actually care about. I am not against markets whose resolution conflicts with common sense if there is a clear understanding that that's what the market is about and will resolve on, and that it is not trying to be a perfect proxy for the underlying event.

As long as the criteria and precedent for when they might be tweaked are clear, it just means people will also be betting on whether specific media outlets will print a false headline, or use language in a way GPT-4 is liable to be misled by, etc. That's not ideal, but honestly neither was a lot of the shaky understanding people had in other shutdown markets - there was lots of uncertainty about how many of them would resolve (though all I was following seem to have resolved sensibly IMHO - this time!)

@chrisjbillington @Charlie Thanks for the words of encouragement. For those concerned, it was indeed an experiment and one I am reluctant to continue.

When I describe the next step, using GPT4/AI to predict these markets that it's both generating and resolving, I find the emotional reaction is quite visceral, not unlike the OPs. Fair enough. If someone wishes to continue what I've done here, all the code (as trivial as it is) is freely available along with the results of these experiments on which they can improve upon.

One small item of note however, I don't see this as a failed market. We learned that NBC likes clickbait headlines, something we can't change after the fact, but we can remove them as a resolution source going forward. We can also perhaps require a consensus between at least two outlets.

I am still skeptical about ignoring ephemeral articles though, as the outlets seem predisposed to rewriting history willy nilly without any accountability. At least there is a record on twitter which they did not delete or issue a correction for, afaik.

It's also worth noting that shutdown markets have a history of controversy as they always surface in strange ways.

I made a slight tweak - "Government Shutdown Official as Congress Misses Spending Bill Deadline"" to "Partial Government Shutdown Official as Congress Misses Spending Bill Deadline""