Will Manifold be more accurate than Polymarket in forecasting the 2024 election?
âž•
Plus
175
á¹€28k
Nov 6
45%
chance

Tracking spreadsheet: https://docs.google.com/spreadsheets/d/1njjIRKNELZ4X_nAbo26_pNEgtUWxcG_cRPGvD_OB1ok/pubhtml

Market inspired by a comment from @MarcusAbramovitch

I will assess a variety of markets pertaining to the 2024 election, depending on which markets are created and have enough users betting on them. Markets that will likely be included (provided I can find real-money markets on these questions):

-Presidential primary markets for Reps and Dems

-Presidential general election market

-Who will win the house

-Who will win the senate

-Who will win particular swing states in the presidential election

-Who will win key senate/house races

-Will the final margin of the election be within a certain range

Any other large markets concerning the election that I can find on both real-money markets and Manifold may also be included. I will likely try to use Polymarket markets, but due to the changing regulatory landscape of real-money markets, I'm hesitant to commit to one real-money prediction site or another. If there are multiple large sites running markets, we can select the one with the largest amount of money / users.

I will select multiple time horizons for each question: for the primary markets, obviously longer time horizons will be needed, as they will converge over the course of the primaries. For most of the other questions, I will likely evaluate at, say, 1 day before, 1 month before, several months before.

I will then generate an R^2 or logloss score for the predictions from Manifold and the real-money markets and compare them. This will be scored like a prediction contest (which I have some experience administering).

If it is clear that Manifold did better than real-money markets, I will resolve YES.

If it is clear that Manifold did worse than real-money markets, I will resolve NO.

If it is truly too close to call (statistical metrics are really trivially different, or different metrics clearly diverge and the final answer is sensitive to small statistical evaluation choices) I will resolve to 50%!

If Manifold or real-money markets for some reason change to not be able to address political questions, or world events lead to a cancellation or restructuring of the 2024 election, or some other crazy stuff prevents this question from being answered, I will resolve N/A.

I will not bet on this market as I think the criteria is subjective. I initially thought that Manifold would be likely to beat real-money markets, but I thought about it more and I'm not so confident, and I'm genuinely interested in learning the answer to this question and will attempt to be as impartial as possible. I will consult with other manifold users on how best to assess and evaluate this question, and will be transparent with how I'm collecting and evaluating the data.

I reserve the right to tweak the description and methodology, especially in the next couple months before any actual data points are revealed, to be receptive of user feedback on the question.

Get
á¹€1,000
and
S3.00
Sort by:

Hey folks! This will take me a while to do accurately (maybe a few weeks) but it's looking increasingly likely this will resolve NO. Polymarket predicted consistently higher odds for Trump for basically every swing state and higher odds for Republicans on senate races, and while Manifold's predictions look quite good (compared with conventional wisdom, etc), Polymarket's look even better.

@benshindel Manifold was also way too slow to react to information on election night

@PlainBG yeah that's pretty typical in my experience just because polymarket has a lot more traders who are a lot faster. I think you can definitely be good at predictions without being good at fast trading though!

sold á¹€44 YES

@benshindel Can you be specific about the time stamp of data used? What time and date will you use as your before election data?

reposted

Hey @traders !!! I'm biased, but I think this is a really cool market with implications on how people use Manifold and Polymarket to benchmark their bets on the other platform, and implications in whether real money improves accuracy over play currency!

I'm gonna try to be more transparent, I've made a spreadsheet to start tracking performance. I will add to this as I feel like it. Feel free to comment suggestions of markets to include. Also feel free to catch any errors I've made. A lot of this relies on roughly averaging by eye the average prediction over 24 hour ranges, for example, so if you think I've done this wrong, feel free to suggest more accurate numbers.

Again, I'm trying my best not to be biased, and hopefully these errors in individual markets by 1%-ish margins won't really affect the overall results.

Don't view these values as final please! I may end up weighting certain markets more than others (I don't think that a market on who wins some house race in Pennsylvania, or something, is as important as a market on whether Biden would drop out, for example). However, I don't want to weight too much, as I do think that including as many markets as possible (and hopefully markets for which partisan lean is less important to avoid correlated error or luck) will yield the best result on determining which platform is more accurate over this cycle. I also may do logloss scoring or another scoring metric as well, not just Brier scores.

https://docs.google.com/spreadsheets/d/1njjIRKNELZ4X_nAbo26_pNEgtUWxcG_cRPGvD_OB1ok/pubhtml

Market with a slightly different flavor to be resolved based on predicted number of Electoral votes.

Since people are trading on this more lately, a few updates:

1) Yes I know Manifold is moving to a real-money model, but it's not there yet and this question will remain the same regardless of if it does, I guess.

2) It looks like Polymarket is definitely the best point of comparison, so unless things change, I will be comparing Polymarket to Manifold.

3) VP selection markets, as well as Biden-dropping-out markets have been pretty deep and interesting, so I'll be including those for sure!

4) Some markets (like the VP choice markets) are extremely volatile, making it difficult to meaningfully select at which time points to evaluate the predictions. I will likely have to make some subjective decisions on how to average values across a day/week/month-long period for these. I'll reiterate: I'm not betting in this market and don't really have a horse in this race so I'll try to be as unbiased as possible to assess this question in the spirit of it! For example, let's say I'm trying to find whether Manifold or Polymarket was more accurate in predicting Trump's VP pick. Let's say I look 1 month out, and for whatever reason, there was a 2 hour spike in JD Vance odds on one platform from 8% to 15% and then back down to 8%, where it sat for another week. I'll probably try to look at the (roughly) average value near that 1 month mark, rather than setting arbitrary time horizons at 12:00am or something.

5) Evaluating the general election race (and swing state outcomes) is obviously not that meaningful from before Biden dropped out, so the time horizons will start after Kamala became the presumptive nominee for those markets, likely.

opened a á¹€44 YES at 52% order

@traders limit order up at 52% if you want to bet NO

opened a á¹€60 YES at 52% order

@voodoo orders are back up. Still figuring out why they got cancelled.

They didn’t get canceled I took them!

I found manifold sometimes doesn’t list the trade when limit orders are fulfilled but I still get the notification.

I had enough up to exit my position.

Think it got confused and thought I didn't have enough balance even though I was exiting my position.

More orders up for you now regardless

Manifold trying to be more accurate than itself after it becomes a real-money market:

@PlasmaBallin Ya, ultimately unsure how to resolve this but I think odds are low that the real-money capabilities are fully realized by the election… but I might have to get creative to answer this in the spirit of the question. I legitimately want to know the answer to this question 😆

@benshindel its still fine by both the spirit and the literal text of the question, because technically manifold will still not be a real money market, it'll be a play money market with some cash prizes.

@benshindel I think if that happens, you can just do it based on whether Manifold is better than other real-money markets. "Being on average better than real-money markets other than itself" is equivalent to "being on average better than real money markets," it's just that the difference is slightly smaller.

@jack Haha true

reposted

Since it looks like this question has come up again, re-upping this

I think Polymarket will beat Manifold this time because Manifold has Trump still lower than he should be.

FYI link to 2022 election analysis (Manifold was more accurate!)

https://manifold.markets/jack/will-manifold-beat-predictit-on-mid

Will Manifold beat PredictIt on midterms predictions?
Resolved YES. I will score several prediction platforms on a set of 10 questions on the outcome of the 2022 US midterm elections. For each prediction platform, I will take the predicted probabilities on Monday evening, and compute the average log score (a measure of the prediction's accuracy) on these questions. This question will resolve as per the title question. "Beat" will be defined as having a better (higher) log score. The question will resolve after all the relevant races are called. The questions and scoring methodology are described here: https://manifold.markets/post/comparing-election-forecast-accurac See related questions here: https://manifold.markets/group/election-forecast-comparison Note that the election results are highly correlated, so the platform that turns out to be most accurate may not have actually been the best set of predictions. The forecast that scores best is probably going to be the forecast that happened to best predict the broader question of how left or right skewed the entire election was, but some of that might be "luck" which it might not be able to repeat across many different election years. To truly measure accuracy well, we'd need to run this experiment several times over different election cycles. I've selected this set of 10 questions: Senate control House control Senate races Pennsylvania - Mehmet Oz R vs John Fetterman D Nevada - Catherine Cortez Masto R vs Adam Laxalt D Georgia - Herschel Walker R vs Raphael Warnock D Wisconsin - Ron Johnson R vs Mandela Barnes D Ohio - J. D. Vance R vs Tim Ryan D Arizona - Blake Masters R vs Mark Kelly D Governor races Texas - Greg Abbott R vs Beto O'Rourke D Pennsylvania - Doug Mastriano R vs Josh Shapiro D Close date updated to 2022-12-15 11:59 am

I think this question will largely be a coin flip on whether the election overall goes more left or more right, because different platforms tend to have different left-right biases. See my analysis at https://firstsigma.substack.com/p/midterm-elections-forecast-comparison-analysis

@jack Yep I understand the limitations completely. I’m hoping to be able to include a broader range of questions about the election that are ideally less correlated (perhaps meta-questions about turnout, demographic swings, etc) but that relies on polymarket having those questions available which I can’t control. I also hope that including multiple time horizons for each question will improve this as well.

@jack I’m definitely open to any methodology suggestions you have since you’ve done this before!

predicts NO

@BenjaminShindel Sure! I think what you wrote in the description is great. Just like you mentioned, one of the biggest limitations in my analysis was finding equivalent forecasts between different platforms. I ended up with a selection of about a dozen markets that were on all the big forecasting platforms - basically the ones you listed except for margin of victory. Would love to look at a broader range of questions if they exist on multiple platforms. More time horizons is great too.

Another thing to be aware of is that the definition of price gets complicated. On manifold it's just one number, but on Polymarket you have a bid-ask spread (the avg of the two is how they define the price) and on PredictIt not only do you have a bid-ask spread but you also have separately priced YES and NO shares.

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules