Meta-Forecasting

dynomight.net/prediction-market-causation

We all want to make good decisions. But it’s hard because we aren’t sure what’s going to happen.

Like, say you want to know if CO₂ emissions will go up in 10 years. One of our best ideas is to have people bet. For example, I might wager my $4 that emissions will go up against your $1 that they will go down. If lots of people bet in a market, the final prices combine everyone’s wisdom and information. If bets were to stabilize at a 4:1 ratio, that suggests that the odds of emissions going up are 4 times larger than the odds of emissions going down, i.e.

, you often need more information. Say someone comes up with a treaty to reduce CO₂ emissions. If you want to know if the treaty will work, you can have people make 

 bets. For example, I might bet my $9 against your $1 with these rules:

If lots of people bet in a market, and prices settle at a 9:1 ratio, that suggests that emissions go up in 90% of the worlds where the treaty gets signed, i.e.

You can run a second market to get the odds of emissions going up, conditional on there 

So here’s our question: Say you do that and the market thinks that emissions are more likely to go up with the treaty than without. That is, say 

. Does that mean that the market thinks that signing the treaty will cause emissions to go up?

People worry about prediction markets for lots of reasons. Maybe someone will manipulate prices for political reasons. Maybe fees will distort prices. Maybe you’ll go Dr. Evil and bet that emissions will go up and then go emit a gazillion tons of CO₂ to ensure that you win. Valid concerns, but let’s ignore them and assume markets output “true” probabilities.

Now, what would explain the odds of emissions going up being higher with the treaty than without? The obvious explanation is that the market thinks the treaty will cause emissions to go up:

Totally plausible. But maybe the market thinks something else. Maybe the treaty does nothing but voters 

 it does something, so emissions going up would cause the treaty to be signed:

In this chain of events, the treaty acts as a kind of “emissions have gone up” award. Even though signing the treaty has no effect on emissions, the fact that it became law increases the odds that emissions have increased. You could still get the same probabilities as in a world where the treaty 

 (albeit with internet points instead of money)

    P[launch | don’t declare] = 5.4%.

 but you can find an implied price using (1) the market for 

 and (3) the ᴘᴏᴡᴇʀ ᴏꜰ ᴍᴀᴛʜ.

. The obvious way of looking at this would be that NATO declaring a no-fly zone would increase the odds of a nuclear launch:

NATO and Russian planes clash over Ukraine

That’s probably the right interpretation. But not 

. For example, do we really know the mettle of NATO leaders? It could be that declaring a no-fly zone has no direct impact on the odds of a launch, but the fact that NATO declares one reveals that NATO leaders have aggressive temperaments and are thus more likely to take 

 aggressive actions (note the first arrow points up):

This could also explain the current probabilities.

Is this starting to sound familiar? If we had a market, I’m confident investors would assign a vastly higher price for 

Is that because reading dynomight makes you more charming and good-looking? Sadly, no—it’s because such people are more likely to value prime-grade autodidactism.

Does this remind you of another issue in probability and statistics? Like, say, the 

 using prediction markets to predict if banned commenters should be reinstated. The idea is that if you get banned, you can write an argument that you will be good and then investors would bet on the odds that Scott would reinstate you if he read your justification, i.e.

This might not work. Suppose that Scott currently pays an assistant to pre-screen the justifications. If the assistant thinks there’s at least an 80% chance someone will be reinstated, they forward their justification to be read, while the rest are ignored. (The assistant gives 

 “ideal” predictions, in the sense that it’s impossible to be more accurate given the same information.)

This distorts the market. Assuming people know about the assistant, the market will give at least an 80% probability of reinstatement to 

, regardless of how bad it is. (You are guaranteed to make money over time by blindly taking any bet with odds lower than 80%. So people will always bid the odds up to 80% or higher.)

What went wrong? The problem is that the quality of the justification influences if it gets read:

             ↓          ↘  

This same distortion of probabilities happens with almost any selection based on the quality of the justifications, though I can’t figure out how to make the general case without some grungy calculations.

But here’s one example: Suppose that Scott looks at only those cases where the prediction market is above 25%. This will also distort the probabilities upward. Suppose that the market is currently at 30% but you think the true probability is 20%. Consider two cases:

That doesn’t seem fair, does it? In the situation where you’re right, the bet is likely to be canceled. So you’ll notice this and not bet. When people who think the probabilities are low drop out of the market, the probabilities get distorted upwards.

The solution to this is to read justifications chosen 

, without any consideration of the quality of the justification or market prices. This breaks the link from the justification to the event of being read:

             ↓             

Look at that: To make things work, all we needed to do was promise to our conditional prediction market that we will do the conditioning 

Where are we? Let’s review. Here’s the typical 

So far, this article has made this argument:

Basically: If you run a prediction market to predict correlations, you get correlations. If you run a prediction market to predict the outcome of a randomized trial, you get causality. But to incentivize people to predict the outcomes of a randomized trial you have to actually 

So that’s the problem. What can we do about it? There are several options.

One option is to think carefully about the system you’re studying. If the causal structure is OK (no reverse causality, no confounders), then you’re OK, and a conditional prediction market will give you what you want. But do be careful, because if you’re wrong you’ll get meaningless results.

There’s an interesting wrinkle here which is that I guess all that you 

 need is that people betting in the market 

 the causal structure is OK. It seems dangerous to try to exploit this, though.

We solved the problem with banned commenters by committing to choosing actions randomly. This option always works. But there’s a cruel irony: The whole point of prediction markets is to make good decisions. If you have to behave 

 to get good predictions, then what are we even doing here?

Fortunately, there’s a good (and well-known) alternative, which is to randomize decisions 

. You tell people: “I will roll a 20-sided die. If it comes up 1-19, everyone gets their money back and I do what I want. If it comes up 20, the bets activate and I decide what to do using a coinflip.”

What’s nice about this is that you can do whatever you want when 1-19 come up, including making your decisions using the market prices. But you must make decisions randomly 

, because if bets never activate, no one will waste their time betting in your market.

This is elegant. Oh, and by the way are you THE NSF or DARPA or THE NIH or A BILLIONAIRE WHO WANTS TO SPEND LOTS OF MONEY AND BRAG ABOUT HOW YOU ADVANCED THE STATE OF HUMAN KNOWLEDGE MORE THAN ALL THOSE OTHER LAME BILLIONAIRES WHO WOULDN’T KNOW A HIGH ROI IF IT HIT THEM IN THE FACE? Well how about this:

Still, I really hope that NATO doesn’t randomize the decision about a no-fly zone.

Consider again the case where people could bet on the odds of a commenter being reinstated. It would be nice to use a market to do filtering, i.e. only read the justifications that have a probability above 25%. But we saw that would distort the market.

There’s an alternative betting structure that could mostly solve this problem. The idea is that if the final market probability is above a threshold you always read the justification. If the final market probability is below a threshold, you only read the justification a fraction of the time, but you make the bets correspondingly larger. For example, we might agree to these rules:

Regardless of if the market is above or below 25%, 

, I will win $1 if I’m right that the person will be reinstated and you will win $4 if you’re right that they won’t. So this market will still incentivize people to bet their true beliefs.

Imagine a prediction market for the odds of a war conditional on who wins a presidential election. You couldn’t interpret this causally. For example, maybe candidate X is less likely to cause a war, but if it already looks like war is looming, then people might vote for candidate Y.

But we can’t randomly pick presidents. (That would interfere with our elections which are 

 just as bad as choosing a winner at random.)

 a market that only activates if the votes are very close to even, so that the outcome is 

This isn’t perfect, because an election being close might influence other variables. For example, imagine that candidate Y likes to undermine the legitimacy of elections. If there’s a clear result it won’t matter, but if X wins narrowly, riots would start and global adversaries might take advantage of the chaos to start wars. In a case like this, a “close call” prediction market might make it look like X is more likely to cause a war, even though X is better under most circumstances.

Still, this is a good idea—it’s not that far-fetched to just assume that things like that don’t happen. You can probably invent other markets with the same principle of “bet that activates if a natural experiment happens to occur”.

Robin Hanson has also suggested “fire the CEO” prediction markets where people bet on the future price of a company, conditional on a CEO being fired (or not). If such a market says the price is more likely to go down if the CEO is fired, that doesn’t necessarily mean the CEO is good because you might imagine something like this happening:

An obvious way to deal with this is to use time: Bet on the stock price 2 years from now, conditional on the CEO being fired in the next month.

This is still no guarantee because it only eliminates reverse causality, not hidden confounders. Maybe everyone knows the CEO is terrible, but hard to fire. But the company was just bought by some billionaire and there are weird rumors that the billionaire hates the CEO and wants to fire them and then drive the company into the ground out of spite:

If you learned that the CEO gets fired soon, you’d have to conclude that the billionaire really does hate them and plans to go on and destroy the company. So even though the CEO is terrible, you’ll have conditional probabilities that make it look like the CEO is good. Time is no magic bullet.

Controlled conditional prediction markets

Finally, one could add control variables to the prediction market. Say you want to know if you should fire the CEO, but you’re worried about things being confounded by a recession. Then you could run four prediction markets for the odds of the stock going up conditioned on each combination of the CEO being fired (or not) and there being a recession (or not). If the CEO truly is crap, you’d expect that

#	Name	Profit
1	Mikhail Samin	Ṁ22,131
2	Davi	Ṁ9,159
3	CatneeBot	Ṁ6,131
4	metabro	Ṁ4,875
5	priestlyt	Ṁ3,692
6	Mira	Ṁ2,402
7	Isaac King	Ṁ2,392
8	tornado	Ṁ1,790
9	GappyV	Ṁ1,391
10	sophia wisdom	Ṁ1,285
11	Edward Kmett	Ṁ1,252
12	Nicodemous	Ṁ1,213
13	arae	Ṁ1,102
14	Jack	Ṁ1,101
15	REVELATE	Ṁ1,045
16	Victor Levoso	Ṁ898
17	Keldon Ahl	Ṁ881
18	Adrian	Ṁ870
19	grecy smith	Ṁ840
20	n99m	Ṁ804
21	Neon Nuke	Ṁ734
22	Frankie	Ṁ720
23	PryingMind	Ṁ709
24	A	Ṁ658
25	Preen	Ṁ629
26	Andrew Conner	Ṁ582
27	Predictor 🔥	Ṁ559
28	Oevrlrod	Ṁ545
29	Jake Hamkvist	Ṁ520
30	Bionic	Ṁ494
31	FRC	Ṁ485
32	lmao	Ṁ481
33	James	Ṁ480
34	firstuserhere	Ṁ472
35	Revel	Ṁ467
36	Yev	Ṁ410
37	Fela	Ṁ382
38	Austin	Ṁ381
39	json	Ṁ372
40	nickten	Ṁ366
41	Svenbonne	Ṁ343
42	ZZZ ZZZ	Ṁ342
43	Arrrrrrr 🤖Bot	Ṁ286
44	David Salzman	Ṁ276
45	Nikita Skovoroda	Ṁ261
46	Franklin Baldo	Ṁ261
47	Jim Hays	Ṁ241
48	Olivia	Ṁ239
49	Fion	Ṁ230

#	Name	Number of traders
1	Isaac King	552
2	levi finkelstein	534
3	GappyV	136
4	brp	78
5	Ammon Lam	70
6	Mikhail Samin	63
7	Yev	58
8	Johnny Ten-Numbers	54
9	Jack	52
10	ZZZ ZZZ	50
11	Joseph Noonan	45
12	MP	43
13	Franklin Baldo	42
14	Adrian	41
15	Fion	40
16	S G	37
17	Josh	36
18	Patrick Delaney	35
19	Olivia	30
20	Electricitypipe	29
21	HMYS	29
22	Manifold Markets	22
23	Yoav	21
24	JAAM	21
25	rockenots	19
26	iteal	17
27	Robin Foster	15
28	Jonathan Ray	14
29	john egbert	13
30	arae	13
31	Duncn	13
32	Simon	12
33	Hei	12
34	Sophia Laird	11
35	Trexmaster	11
36	aps	10
37	Jonny Big Boy	10
38	VivaLaPanda	9
39	tailcalled	9
40	Nico D	9
41	Alex L	8
42	Paul Hobbs	7
43	Big Chungus	6
44	Hamm	6
45	firstuserhere	6
46	Noah	5
47	PS	5
48	Brian T. Edwards	4
49	Jim Hays	3
50	Gavin McCarthy-Bui	3

Latest Posts

🏅 Top traders

🏅 Top creators