Nate Silver, the founder of FiveThirtyEight, recently was laid off (it has been a part of ABC for the last decade). According to him, "Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney. The license term for these models expires with [Nate Silver's] contract this summer. [He] still own these models, and can license or sell them elsewhere" (https://natesilver.substack.com/p/some-personal-news). If he releases predictions for the 2024 US elections, will they be more accurate than the model put out by FiveThirtyEight?
If either Nate Silver or FiveThirtyEight do not release election predictions for 2024, resolves N/A. If Nate Silver continues to license his model to Disney/ABC/FiveThirtyEight, resolves N/A. If Nate Silver licenses his model to some group other than Disney/ABC/FiveThirtyEight, that will count as "his model".
The predictions used will be the final predictions released before the elections. Any predictions given earlier than that will be ignored, even if they are more accurate.
If multiple models are used (such as a "Deluxe Model" and a "Classic Model"), it will be whichever model is more prominent in publications. If multiple models seem to be equally prominent, it will be the model that includes more information.
I'll define "more accurate" by scoring both predictions with a "log score" method, as detailed in this Substack post (by @jack), weighted so that a Senate seat is worth 12.8 House seats and the Presidency is worth 435 House seats. This allocation is so that the House, Senate, and Presidency are all weighted roughly equally. If a model does not call a specific seat, that will count as a 50/50 prediction. A prediction phrased ">99%" or "<1%" will be treated as 99.5% and 0.5% respectively.
Related questions
Nate Silver just announced that he will be doing a 2024 Presidential model, but not Congressional. Do you think this should:
Resolve N/A?
Only count the Presidency predictions?
Count all House and Senate seats as 50/50 predictions for Silver (this market will almost definitely resolve NO)?
Leave it as-is and re-ask, hoping that he changes his mind?
I'm going to put comments for each option, and I'll go with whichever option gets the most likes.
https://www.natesilver.net/p/the-election-model-is-a-little-too/
Nate Silver is saying that
He might not release an election model for 2024
He might do a lo-fi model, with less frequent predictions or rounded predictions
He might sell or license it for private usage (eg. by financial firms looking to hedge risk)
In the first and third case, this would resolve N/A (since there’s no model that I would have access to judge). In the second case, I would try to compare them as similarly as possible. However, if they’re too dissimilar and meaningful comparison isn’t possible, I would resolve N/A.
To be clear, I support Silver in whatever choice he makes. It’s ultimately his personal decision and he’s completely correct about a lot of the negative effects of his model (both on himself and a lot of the public).
@Gabrielle In the past, 538 has described races as having a "<1%" chance of having a certain outcome. How will you treat predictions of this form?
@Gabrielle My inclination would be 0.5%, but your way makes sense too. If I make any markets to arb against this I'll lean towards following whatever you do.
This is one of those weird meta-forecasting questions where it's less about "how good are the respective forecasters" and more about "what is the chance that Silver will be more uncertain about the outcome XOR what is the chance that the favored party the day before the election will outperform expectations."
@BoltonBailey Relevant question: If we backtested this on the 2020 election, where democrats underperformed polls but Biden was still elected, would a predictor who was slightly more bullish on democrats have been more accurate according to this metric?
@BoltonBailey If anything, maybe this market should be at less than 50% on the grounds that Silver is likelier to be more uncertain, but that whatever 538 and he both predict is likelier will come true.
@BoltonBailey In 2020 Silver predicted 90% chance Biden would win and Morris predicted a 97% chance he would win.
You should judge the models by how well calibrated they are, not whether they "correctly" predict binary outcomes. For example: https://firstsigma.substack.com/i/85825582/scoring
@Gabrielle Yeah, I think Nate’s will probably be “better” (85%), but in any given election, there’s correlated variance, so that’s why I bet NO.
@cos I don't think calibration is the only factor. If I predict every seat at 50% my calibration will be great, but I won't be a good predictor.