Brian T. Edwards

Predictions on Predictions

https://firstsigma.github.io/midterm-elections-forecast-comparison

It's interesting to compare forecasts between different prediction platforms, but it's rare for them to have questions that are identical enough to compare easily. Elections offer one helpful opportunity.

I will score several prediction platforms on a set of 10 questions on the outcome of the 2022 US midterm elections: Senate and House control, and several individual Senate and gubernatorial races with high forecasting interest.

For each prediction platform, I will take the predicted probabilities on Monday evening, and compute the average 

 on these questions. This is a measure of prediction accuracy - higher log score means better accuracy.

I plan to compare these prediction platforms:

Others can be added, just add a comment with the data on their predictions for each of the questions above.

: the election is much closer to one overall prediction than a set of independent predictions, because the races are highly correlated. The forecast that scores best is probably going to be the forecast that landed closest to the mark on the broader question of how much the nation overall went left or right, or how far left or right the polls were biased - and a large part of this is chance. So despite the large number of individual races, each election cycle can be thought of as roughly one data point, and to truly measure accuracy well, we'd need to run this experiment several times over different election cycles.

Of course, I've also created meta prediction markets on which prediction platform will be the most accurate: 

https://manifold.markets/group/election-forecast-comparison

I selected this set of 10 questions to compare across prediction platforms:

These were selected as races that had a high amount of interest across the prediction platforms. The main reason for using a limited set of questions is that not all prediction platforms made forecasts on all races - the main limiting factor was which questions were on Metaculus. (I did later find a couple more races on Metaculus, but did not add them to my list because I had already preregistered the question set.) Using a smaller set of questions also makes the data collection easier for me.

They are not all highly competitive races - which is a good thing for looking at how accurate and well-calibrated predictions are across a range of high or low competitiveness races.

See prediction questions on which platforms will be most accurate here:

 https://manifold.markets/group/election-forecast-comparison

How accurate are different prediction platforms on the US 2022 midterm elections?

Comparing election forecast accuracy

Isaac King

Jack

Ammon Lam

Bionic

ZZZ ZZZ

Tomek K 🟡

Conflux

Ibozz91

Manifold Markets

Franklin Baldo

Johnny Ten-Numbers

arae

Matt P

Electricitypipe

Undox

Tim P

Charlie

Bolton Bailey

Lifejacker

N.C. Young

Olivia

Benjamin Ikuta

Daniel Reeves

JAAM

Vincent Luczkow

Martin Randall

Forrest

GeorgeVii

Simon

Carson Gale

Grant Thompson

Trexmaster

Enopoletus Harding

Edward Yu

Austin

zoli

Sophia Laird

squirrelfish

Gurkenglas

Andrew G

footgun

Adam

Jake Wasserman

Collected Over Spread

Gigacasting

Jack2

Catnee

Michael Wheatley

Katja Grace

bingeworthy

James

REVELATE

Arbitrage Bot

Mitchell Roby

Steven

yaboi69

Ian Philips

Joshua

SneakySly

Adrian

Benjamin Cosman

Gabrielle

HoosierDaddy

Alice

girlboss

Preen

Breeze ice

Dr P

Tripping

Nathan Young

Thekla

Mira

Yaakov Saxon

Marty Dettmann

trainbot

joy_void_joy

Scott Lawrence

#	Name	Profit
1	Jack2	Ṁ9,420
2	CatneeBot	Ṁ5,239
3	Michael Wheatley	Ṁ3,749
4	Gurkenglas	Ṁ3,444
5	Katja Grace	Ṁ2,432
6	Yev	Ṁ1,959
7	0o0	Ṁ1,235
8	bingeworthy	Ṁ1,225
9	James	Ṁ1,193
10	ZZZ ZZZ	Ṁ869
11	REVELATE	Ṁ858
12	Arbitrage BotBot	Ṁ807
13	Mitchell Roby	Ṁ761
14	Daniel Reeves	Ṁ681
15	Gigacasting	Ṁ656
16	Bionic	Ṁ649
17	Steven	Ṁ645
18	arae	Ṁ617
19	FRC	Ṁ611
20	Johnny Ten-Numbers	Ṁ451
21	yaboi69	Ṁ445
22	Ian Philips	Ṁ391
23	Joshua	Ṁ356
24	Austin	Ṁ352
25	SneakySly	Ṁ342
26	Ammon Lam	Ṁ312
27	zzq	Ṁ303
28	Adrian	Ṁ299
29	GeorgeVii	Ṁ261
30	Benjamin Cosman	Ṁ243
31	Gabrielle	Ṁ233
32	HoosierDaddy	Ṁ220
33	Alice	Ṁ219
34	girlboss	Ṁ212
35	Preen	Ṁ201
36	Breeze ice	Ṁ194
37	Dr P	Ṁ187
38	Tripping	Ṁ162
39	Nathan Young	Ṁ152
40	Thekla	Ṁ151
41	Mira	Ṁ145
42	Yaakov Saxon	Ṁ143
43	N.C. Young	Ṁ141
44	Marty Dettmann	Ṁ125
45	Carson Gale	Ṁ125
46	trainbotBot	Ṁ119
47	joy_void_joy	Ṁ119
48	Tim P	Ṁ113
49	Scott Lawrence	Ṁ113
50	Tomek K 🟡	Ṁ113

#	Name	Number of traders
1	Isaac King	754
2	Brian T. Edwards	327
3	Jack	318
4	Yev	272
5	Ammon Lam	97
6	S G	66
7	Bionic	63
8	ZZZ ZZZ	62
9	Tomek K 🟡	56
10	Conflux	54
11	Ibozz91	48
12	Manifold Markets	44
13	Franklin Baldo	43
14	Johnny Ten-Numbers	39
15	arae	33
16	Matt P	31
17	Electricitypipe	29
18	Undox	27
19	Tim P	26
20	Charlie	25
21	Bolton Bailey	25
22	Lifejacker	23
23	N.C. Young	20
24	Olivia	20
25	Benjamin Ikuta	20
26	Daniel Reeves	19
27	JAAM	18
28	Vincent Luczkow	15
29	Martin Randall	14
30	Forrest	12
31	GeorgeVii	12
32	Simon	12
33	Carson Gale	12
34	Grant Thompson	12
35	Trexmaster	11
36	Enopoletus Harding	11
37	brp	11
38	Edward Yu	10
39	Austin	8
40	zoli	7
41	Sophia Laird	7
42	squirrelfish	7
43	Gurkenglas	5
44	Andrew G	5
45	ian	5
46	footgun	4
47	Adam	4
48	Jake Wasserman	2
49	Collected Over Spread	1
50	Gigacasting	1

Latest Posts

🏅 Top traders

🏅 Top creators