Resolved: That it is bad to resolve markets as N/A and that it is usually — exceptions discussed below! — worth the effort to untangle ambiguities and hash out what's fairest to get a proper resolution. That can be exhausting, invidious work and it's understandable if you're not up for it, but at least consider it a last resort to resolve N/A. Critically, if traders are lobbying for an N/A resolution, that might, for sometimes subtle reasons, be all the more reason not to do it.
But this is easier said than done. Sometimes you pose an innocent-seeming question and traders make different assumptions and bet accordingly. Later, the ambiguities come to light and all possible resolutions feel grievously unfair to one set of traders or another. And it's possible that N/A is what's least unfair. But here's a collection of reasons to resist resolving N/A, followed by a list of principled exceptions, when N/A is OK.
1. It's anticlimactic and throws away people's work
This one seems obvious and seems relatively minor, but is worse than it seems. Resolving N/A unwinds everyone's trades as if they never happened. That includes clawing back profits of those who correctly predicted a price movement and then cashed out. This invariably surprises people. It's also messy for Manifold if clawbacks make traders' balances go negative. It's even a vector for abuse.
(Separate can of worms: Could Manifold just not do that? Why not unwind everyone's positions but not unwind cashed-out profits? Well, that creates an even bigger vector for abuse. Point being, N/A resolutions are messier than they seem.)
2. It spoils incentives
This is kind of a corollary to reason #1 but is also a counterargument to a pro-N/A argument. Namely, that the real value of a prediction market is to see the market probability evolve. The final resolution, or lack thereof, doesn't undo that value. That's true but resolving N/A sets a bad precedent, making people less motivated to do the work to make market probabilities accurate in the first place if they think N/A is at all likely.
3. It can be a way to cheat
Now we get to the more serious and subtle problems with N/A. As an extreme example to illustrate how bad this can be, imagine creating a "Will aliens arrive by Christmas?" market. You dump all your money on YES. If aliens show up, resolve to YES, win big. If aliens don't show up, you start stirring up fairness concerns. It wasn't specified whether you meant extraterrestrial aliens. Et cetera. Get the debate going and then suggest that maybe the fairest thing is to resolve N/A. Everyone gets their money back, no harm no foul. But you totally rigged it! You made a big bet on YES and then found a way to weasel out of paying up when you were proven wrong.
It's not normally so blatant but it's worth having that in mind. "Just give everyone their money back, no harm no foul" can be extremely incorrect. Totally harm, totally a foul. Be skeptical of anyone suggesting that resolving N/A is what's fairest. Would they be making that argument if reality had played out a different way? Their seemingly reasonable arguments for an N/A resolution might effectively be trying to get out of paying up on their bad bet.
4. It can distort market probabilities in decision markets
Say you want to know whether, if you buy Apple's VR headset, you'll actually use it enough that you'll be happy you bought it. So you make a market asking exactly that. You might naturally commit to resolving N/A if the market convinces you you'd regret the purchase. No purchase, no way to resolve the market. The problem is that this can significantly bias the market's estimate towards YES.
Imagine I'm pretty sure you'd regret getting the VR headset. If I drive the price down to my true estimate of, say, 10% that you'll like it, my upside is cut off. If I'm wrong and the price goes back up and you buy the thing and have no regrets, I lose money. If I'm right then you defer to the wisdom of the market, don't buy the thing, and resolve N/A. I'm paying up if I'm wrong and getting no payout if I'm right. Thus I won't drive the price down enough to convince you not to do the thing. I might not bet NO at all, in case others drive the price down that far.
(Separate can of worms about what you might do instead: You could resolve to the market price, using resolve-to-PROB, in the case that you don't buy the headset. The risk with that is market manipulation. Driving the price to 1% is guaranteed profit if I think I have enough money to keep the price there. At that price I know you won't buy the headset and thus you'll resolve to the market's estimate of a 1% chance of headset happiness. Ka-ching. My own meta-prediction is that that's generally too risky to try to exploit in practice. Suggestion: don't commit to a particular threshold for doing vs not doing the thing. That way any would-be manipulator has to worry that you might buy the headset despite the market seeming 99% sure you'll regret it. If you do, and don't in fact regret it, the manipulators lose their shirts. Or you can just use some non-market estimate of the probability that you would've regretted buying the headset. Resolve-to-PROB at that estimate and the incentives to manipulate the market are gone. How to find an unbiased outside probability is a sub-can-of-worms.)
5. Ambiguous outcomes are often better resolved to a probability
Did the thing being predicted kinda/partially/ambiguously happen? You don't have to endlessly debate YES vs NO and give up and resolve N/A when neither side can claim a decisive victory. Instead the creator can make a judgment call like "this counts as 75% YES" and just resolve to that. Resolve-to-PROB, as Manifold calls it. It may seem weird but is pretty exquisitely fair if that 75% or whatever fraction is the best reflection of the weird reality that ended up happening.
(The can of worms again: It's tempting to use the market probability itself as the PROB to resolve to when there's no consensus on the ground truth. After all, what is the market probability but the aggregated opinion of all participants? And this can work but is fraught. You have to somehow know that you're not opening the market up to manipulation. And there are ways to be reasonably sanguine about that. See the can-of-worms parenthetical on reason #4 for more.)
Exceptions
Here are all the principled exceptions I know of. If you have others or if you have an example of a market that you think should resolve N/A but doesn't fit one of these templates, let me know!
A. Nonpredictive questions
Like "Is Trump good?". If nothing is even being predicted then it might as well resolve N/A.
B. Conditional markets
It's fine if the resolution criteria explicitly stipulate an N/A resolution if such-and-such happens. Like "Will Biden beat Trump? Resolves N/A if either of them dies before the election."
C. Self-contradictory resolution criteria
This is an easy mistake for market creators to make. Maybe they forget a previous clarification and effectively change the resolution criteria after some people already bet. If all possible resolutions would be unfair to some subset of traders, resolving N/A may be least unfair.
D. Pivotal ambiguity that's noticed too late to clarify
Ambiguity itself shouldn't be grounds for N/A. Traders should just ask the market creator clarifying questions (and trade with caution until they get the answers). But sometimes, as in exception C, the creator ends up backed into a corner where any resolution would be unfair to some traders.
How can that happen without a direct self-contradiction? Suppose a market is created with unanticipated ambiguity. Say there are 2 different interpretations of the question, A and B. For example, imagine a creator asks, "will X have happened by time T?" and meant "will X be the case at time T?" but maybe a more natural interpretation is "will X happen at any time before time T?". Some traders trade assuming A and some trade assuming B. Critically, both sets of traders are being reasonable in assuming A (B) and not thinking of possible interpretation B (A).
Or maybe one interpretation wouldn't have been reasonable to presume per se but is the one the creator had in mind, so, ipso facto, it is. (This shades into exception C, if the creator is effectively contradicting themself with their clarifications.)
If we're lucky, interpretations A and B will both turn out to yield the same resolution. That's nonpivotal ambiguity. Don't resolve N/A in that case! Wait it out in case the ambiguity turns out not to matter.
If we're unlucky, A and B yield different resolutions and we're screwed. Both possible resolutions would be unfair to some traders, so we have to resolve N/A.
Again, flagging an ambiguity you should've asked about before investing heavily in YES or NO is not grounds for N/A! That opens the door to cheating where traders ignore an ambiguity and only raise the objection if the market moves against them.
We can't trust market probabilities if traders can make the market more or less likely to be voided depending on how reality plays out.
(I originally said some of this in a "will DALL-E be able to draw blue grass and a green sky?" market and a "will I regret bleaching my hair?" market. It was in the latter that user @Conflux pointed out the potential for market distortion in a "will I regret XYZ?" market. For more on trickinesses with decision markets, see Prediction Market Does Not Imply Causation.)
1,000People are also trading
Case study from @Primer: https://manifold.markets/Dariusn9gL/will-roblox-die-in-2027#khoi0wsye6o
Recap of my thoughts:
In an underspecified market, traders should understand that it's incumbent on them to clarify before trading and that we defer to the market creator.
Even so, it can be ok to trade in an underspecified market, caveat emptor. In the above case study, maybe you think it's at least a 90% chance that Roblox has steady growth through 2027. Then buying "NO, won't die" is a good trade at 20% regardless of the definition of death the market creator intended.
My latest reply:
Is dilution of markets bad? It's not obvious to me. As you say, attention and mana are finite. On the other hand, suppose that instead of intervening to N/A an underspecified market, the protocol was to add a link to a better-specified market. Maybe that ends up directing more users to canonical markets than would otherwise see them?
But I think you're saying that some markets are just negative value-add for Manifold, by being too agonizing to resolve and/or by creating the conditions for bitter disputes about fairness.
I guess I'm skeptical of that, or at least I worry that the meta problem of deciding when a market's value has gone negative is even thornier. One might even worry that this could be another avenue for traders to try to cause an N/A resolution in order to avoid paying up on bad bets.
To get more concrete, would you advocate for the following protocol, for example: "any market with a blank description and a creator who hasn't replied to requests for clarification for 30 days is automatically resolved N/A"? I have to think harder about whether I'd endorse that. Are we sure there's no vector for abuse, like the creator themself trading and then causing the N/A to happen or not depending on how successful their trades have been?
@dreev I don't really advocate for proactive policies, just for fixing incentives. If ambiguous markets usually resolve N/A, that should do a lot to fix incentives. Being proactive would help though.
Let's take the "Will Roblox Die in 2027?" market: Do you think it is positive value?
We wouldn't be able to draw the perfect line. But we don't have to. Catching some fraudsters is enough to prevent everyone from simply not paying taxes.
I know you're concerned with people, mainly creators, trading in an ambiguous market and then nullifying their losses by N/Aing. I'd argue:
If this was a widely used vector of abuse, I'd still prefer it over ambiguous markets desperately needing to resolve non-N/A
If bad markets get N/Aed, the vector of abuse stops working, as people will stop trading there
This doesn't really happen. I've been around long time and so far I've seen a single situation where this could have happened, but in that case the loophole was introduced accidentally and not originally intended to cut losses (can't find it now, something about loans, had @-ed you some days ago).
A much more concerning vector of abuse is already existing. Everyone can create an ambiguous market, bet on it and resolve however they like (maybe after adding clarifications).
Let's take the "Will Roblox Die in 2027?" market: Do you think it is positive value?
I think so? Nonnegative value at least? Or at least in expectation, since the market creator may come back and answer questions. But it's not obvious that it's negative-value regardless.
I know you're concerned with people, mainly creators, trading in an ambiguous market and then nullifying their losses by N/Aing
In the writeup for this poll I gave an example of a market creator cynically creating and betting in an ambiguous market in order to have an excuse to N/A it if their bets go badly. But I didn't mean to imply it was mostly about market creators. I'd say the opposite. See my "bold claim" comment. Like in this Roblox case, I'm imagining a trader betting "NO, won't die" on the theory that even if Roblox goes bankrupt, the brand will be sold off and there will still be something called Roblox and since the market was so ambiguous it should be N/A'd. Or the other way around: buying "YES, will die" and then arguing that competitors gaining market share should count and, again, since the market was ambiguous it should be N/A'd.
This doesn't really happen.
I agree the blatant premeditated version doesn't happen (much?). I just meant that as an intuition pump for why "give everyone their money back, no harm no foul" is wrong. But bullying market creators into N/A'ing in order to avoid losses has felt like a problem. Maybe we're making progress on it. My motivation with my anti-N/A campaign here is to help fight against it.
(I should emphasize that I know we're both motivated by making Manifold better and aligning incentives for good-faith markets and good-faith trading.)
A much more concerning vector of abuse is already existing. Everyone can create an ambiguous market, bet on it and resolve however they like (maybe after adding clarifications).
Ok, yeah, this is a problem. We have a lot of ongoing discussions amongst the moderators about what to do about this. For example, there's a thread in the #moderators channel and @Gabrielle has an excellent proposal. (And more recent things I've lost track of.)
What if we consider the case of a blatantly bad-faith market? Even then I worry about N/A'ing as a solution. It gives both perpetrators and their victims all their money back. Which means the perpetrators are incentivized to do as much crime as possible, if it either succeeds or is simply nullified. Maybe N/A'ing the market and imposing fines is a fair enforcement strategy? I think I'd rather see a bad-faith market usurped and run fairly, only N/A'ing it in the case of the principled exceptions to the "no N/A'ing" principle that I enumerated in the writeup for this poll. But if there's another principled exception involving bad-faith markets, let's add it.
Back to the Roblox case study, I just noticed that the creator bet the price up to 99% after creating the market. That does look like bad faith, but could also just be bad epistemics. I like @Gabrielle's solution of making the creator's conflict of interest more obvious to traders. I still don't think I like preemptive N/A'ing in a case like this.
@dreev I feel bad about picking the Roblox market as an example, as it turned out to pretty much totally prove my point, but that's just pure luck! In practice, most situations are not near as clear.
Like in this Roblox case, I'm imagining a trader betting "NO, won't die" on the theory that even if Roblox goes bankrupt, the brand will be sold off and there will still be something called Roblox and since the market was so ambiguous it should be N/A'd. Or the other way around: buying "YES, will die" and then arguing that competitors gaining market share should count and, again, since the market was ambiguous it should be N/A'd.
Yeah, I agree this is possible. But just as N/A can be argued, either Yes or No can be argued. And I posit those are almost strictly worse:
Happens way more often
Does way more damage: Good faith traders lose their Mana
Sets wrong incentives: Creator gets to keep bonuses
@Primer Debating YES vs NO is normal and expected (though I prefer norms of deference to the market creator). What's more insidious about N/A is the counterfactuals. If reality goes your way, no debate, just let it resolve. If not, start the debate and settle on N/A as if that's a fair compromise that gives all the good-faith traders their money back. It does give everyone their money back, ex post, but ex ante those traders got scammed. They made a bet with, unbeknownst to them, downside and no upside. See what I mean?
That's the value of having a principled stand against N/A resolutions, modulo the list of principled exceptions.
@dreev Hard disagree: I'd say debating Yes or No is not normal. It should hardly ever happen, and if it happens it's very likely something went terribly wrong.
But yes, I see what you mean. We differ on priorization. I see more value in protecting innocent traders and setting incentives for better markets, you see more value in preventing malicious actors and protecting some subset of the innocent traders.
I'd say debating Yes or No is not normal. It should hardly ever happen, and if it happens it's very likely something went terribly wrong.
Ah, yes, I do agree (see also my parenthetical about deferring to market creators) and I think what I'd like to argue is that it's better to have the loud failure of an ambiguous market that gets debated than the quieter failure of N/A'ing the market.
Another stab at making the argument: If YES traders and NO traders are fighting it out, at least the incentives are balanced. But if the losing side is arguing for N/A, the deck is stacked. You can present a principled-sounding argument for N/A and no one sees that you wouldn't be making that argument if you had bet differently. Invisible counterfactuals are sneaky like that. There's even a kind of concentrated-benefits/distributed-harms problem: no one has enough incentive to argue against N/A since everyone gets their money back in that case. There's an asymmetry where people who stand to lose with a normal resolution argue hard for N/A but the people who stand to gain with a normal resolution don't argue against it. I guess partly it seems petty to object to getting your money back?
It reminds me of the scam where someone sells a nonexistent thingamabob, takes people's money, and then for everyone who complains they apologize profusely and give them their money back. Free money from the subset who fail to complain! And little incentive to punish the scammer since everyone who noticed anything amiss was made whole. That's part of the intuition for how, if people create or trade in scammily ambiguous markets, it's not enough to N/A the market.
Not that taking N/A off the table (modulo the principled exceptions) makes scamming impossible. But it raises the bar on how blatant you have to be. Like if (hypothetically) the Roblox market is a scam, the creator is going to have to say "I'm resolving this YES, Roblox died" and then all the NO traders can cry foul and the creator can be asked to explain themself, etc. Hopefully a moderator can override the resolution and the would-be scammer loses their shirt (hypothetically).
Maybe the most pragmatic question is which is more work for moderators?
it's better to have the loud failure of an ambiguous market that gets debated than the quieter failure of N/A'ing the market
We could always just @ each other and the N/As will be loud enough 😜
@dreev Bad markets get the best attention on manifold. Long-term markets with clearly defined criteria that would actually be useful for predicting the future get 50 traders on a really good day.
Markets that are all about arguing what the question means get all the attention. Here's an example:
Another one of my favorite examples of manifold degeneracy is this nonsense:
Well, maybe manifold is just really into Dwarkesh? Not particularly, a reasonable market about his guests has under 2% of the traders.
@ProjectVictory I agree about this. In Discord we've been talking about how to fight this problem. One idea is a moderation option for de-promoting a market that's less extreme than unlisting it (or N/A'ing it).
Bold claim: The worst thing about Manifold is when traders rules-lawyer the resolution criteria to find a creative way to justify a YES or a NO, go all in on it without getting clarification, and then try to bully or jawbone the market creator into accepting their interpretation. If that doesn't work they'll suggest N/A as a seemingly fair-minded compromise. That's often an appealing option for the market creator, who's exhausted by the arguing. But of course the rules-lawyers have still successfully cheated with an N/A resolution. They bet based on their interpretation of the market and then weaseled out of paying up when they were wrong.
Besides resisting N/A resolutions, I think the best way to avoid that is strengthening the norms of deference to the market creator. If you spot a clever loophole or something, ask the market creator before betting on it. Collaborate with the creator on shaping the resolution criteria to align with the spirit of the question.
One related thought - I think there is a temptation to N/A when the outcome as defined by the literal terms of a question contradict the spirit of the question.
I’m having this debate now with some AI related questions. Often they say some version of “Will AI model achieve score on this leaderboard?” The AI clearly achieves a score on the underlying benchmark….but there are lots of abandoned and no longer updating leaderboards.
I think market creators and participants think that literal terms are safe and good. They often are, but if you hold that view you have to own the downside of “thing X clearly happened but official source Y stupidly doesn’t say it happened in time”.
@MRME When a spirit/letter divergence like that happens the traders need to ask the creator the clarifying question and let the creator make the call (I prefer spirit over letter myself but it may depend on the wording, how clear spirit and letter each were). Important not to N/A in cases like this!
If, as a trader, you presume letter trumps spirit (or vice versa) and the creator disagrees and you lose your shirt, you've no one but yourself to blame. Trying to abrogate your loss by lobbying for N/A should be viewed as tantamount to cheating.
You've done a lot over the years to reduce my reliance on N/As and I think this is an excellent treatise. I'm still not a "Never N/A!" camper, sometimes convenience really is just worth it and a world where we CAN resolve N/A is orders of magnitude better than one where it's not even a possibility, but these are great baseline considerations.
When I'm wearing my mod hat I have to trust traders to give me honest answers that benefit them. They are generally reliable to do so, and it's pretty obvious when they're off that baseline (either more neutral or more resistant). And sometimes they don't consciously realize they've doubled down so hard on a given resolution (like N/A) for conflict-of-interest reasons that emotions get involved, but I think that's EXACTLY when these kinds of rules and exceptions need to be weighed. It's a good lesson for non-mod-hat times, too, imo, whether as a creator or even as a trader.
I've gone even further to try to solve some of the problems with N/A. One thing I posited when I was designing for Playmoney with @case-and-team in my "pirate.markets" proposal was the idea of a "slush fund" (which was on theme but could easily have a myriad of other names). The slush fund would be a side bet directly against the market's liquidity that only pays out in the event of an N/A result, at the risk of simply being evaporated in any other resolution. It serves a dual-purpose: (1) as a way to disincentive creators from relying too much on N/A and (2) as a rating to signal market confidence, since a low slush fund in a highly traded market most likely indicates either robust resolution criteria and/or a strong likelihood of an uncontroversial result. Would love to see Manifold adopt something like that!
(The slush fund is just the tip of the iceberg of course. I've got more designs and models for dayyys. At some point they need to just hire me to revamp the economy here, but that's a completely different tangent)
Notes from a case study a while back in which @strutheo was inclined to N/A his market about the number of open limit orders at the end of 2024:
Opening salvo: This is a fundamentally answerable question and depending on how people bet, there could be incentive to put in the work to answer it. To make it more stylized, imagine Alice bet heavily on YES. Then reality plays out and it looks like Alice is wrong. But also Alice is the one most qualified to find out the answer. Her incentive is to sit on her hands and get her money back with the N/A resolution. If reality had played out in her favor, she'd have done the work and gotten her YES payout.
So N/A is potentially unfair. But what can we do about it, without anyone volunteering to do the work to get the true answer?
I think resolve-to-PROB at the last market probability is non-crazy. The danger in that is market manipulation, but (a) we don't think there was market manipulation here, and (b) the incentives for market manipulation are low in a case like this. Too much risk of resolving to the ground truth for manipulators to want to try anything, I imagine. (I've been known to be pretty naive about this kind of thing though!)
Another option would be to just leave it open? I think people hate that. With loans, I think it's no skin off people's noses, but I may be missing some considerations here too. Like just tidiness -- it feels gross to have markets unresolved long after the truth is knowable.
(I don't mean to be persnickety about @strutheo's market in particular. This general question is just academically interesting to me.)
Another example, from Metaculus, where N/A is the best choice in my opinion, as well as in the opinion of Metaculus admins:
Lots of good discussion in the comments.
@Primer Great example and I agree with Metaculus's decision in that case. I think this falls under exception C in my little essay here. "Sometimes you pose an innocent-seeming question and traders make different assumptions and bet accordingly. Later, the unanticipated ambiguities come to light and all possible resolutions feel grievously unfair to one set of traders or another. And it's possible that N/A is indeed what's least unfair."
EDIT: What I'm describing above is more like what I'm now calling Exception D.
To spell out my understanding of the reason for the partial N/A in that Metaculus example: The data source they chose for the USD/CAD exchange rate was wrong. Some traders were trading based on that data source and others were trading based on the true exchange rate. Both sets of traders were making a reasonable assumption and probably no one even noticed the discrepancy until later so there was no clarifying question they could've asked. So that's an irreconcilable ambiguity that kind of forces an N/A resolution.
I don't think that underspecified Manifold markets (like "Will we get AGI by 2027? By AGI I mean very smart robots." or whatever) have a similar justification for N/A resolution. I think to qualify for this exception to avoiding N/A, it has to be the case that any other resolution would be fundamentally unfair to some set of traders. If a trader gets hosed because they ignored an obvious ambiguity, that's not fundamentally unfair. They should've clarified before trading.
So my contention is that to argue for exception C [or D] you need two sets of traders to each make different assumptions and for both assumptions to have felt so natural that they can't reasonably have been expected to think to clarify.
That's exactly what happened in the above Metaculus market. A common way for it to happen on Manifold is when the creator effectively changes the resolution criteria, like by disambiguating something that some traders reasonably thought was unambiguous in the other direction.
So my contention is that to argue for exception C you need two sets of traders to each make different assumptions and for both assumptions to have felt so reasonable that they can't reasonably have been expected to have checked them.
Who would be able to judge that?
Example: The "Will this Yudkowsky Tweet hold up 2024 version" market. When I first traded there, I had no doubt the market was asking whether the tweet would be regarded as a good take in 2024, as in "someone says x will happen in 2026, it's 2024 now, but it seems quite possible he will turn out right, so the tweet holds up". At least some other traders shared this reasoning, there was a comment early on.
I guess you wouldn't have joined me in arguing for exception C, because for you, my reading wasn't reasonable. So this exception C is good on paper only. In practice it comes down to how many rich traders have their Mana on which side, who is more influential, how did mods themselves interpret the market, etc.
Nobody is gonna advocate for N/A on behalf of 5 small traders who might have invested 30% of their balance on their reasonable interpretation, when 100 traders interpreted the market differently.
So you got my Yes for exception C, but it should be taken seriously and actually be in use. It's too easy to counter a "Huh, I had been interpreting this clearly as X" with a "Well 'duh, it's Y. You should have asked for clarification before trading.".
@Primer Fair point, we don't want a minority (as measured in people or in mana) to get bowled over by a majority. I also share your aversion to argumentum ad lapidem, appeal to ridicule, and related rhetorical ploys where people try to create a false sense of consensus by loudly saying how obvious something is.
So, yeah, we could say that if any two groups of traders make opposing credible claims to have made a good-faith assumption that they couldn't reasonably have anticipated needing to clarify -- even if one "group" is a single trader -- that's enough to at least put N/A on the table as possibly the fairest resolution.
Sounds like we may be gradually converging? So far I'm holding firm that blatantly underspecified markets shouldn't be ipso facto N/A'd as long as they were created in good faith to ask a legitimate question about the future.
@dreev Don't want to get this mixed up with my earlier comment, but this is another interesting example: /RemNi/will-we-get-agi-before-2026-3d9bfaa96a61
The creator is gone, I left some thoughts in a comment over there, but it's part of a whole series of markets all suffering from the same problem. The 2025 version didn't create much drama, but some year will. I suppose you wouldn't want to N/A? What course of action would you take, follow your suggestion of "adding the clarifications as they come up"? It'd be up to you or one of your fellow co-mods to do that right now.
@Primer I think I agree with your points there. I'm mulling it. My first thought is, can we just declare that we'll follow Metaculus's resolution? If that's acceptable to traders maybe it nips all potential drama in the bud.
(So far it sounds like people are objecting to N/A'ing those markets simply because they've put work into trading, maybe sometimes making significant profit from trading on the volatility. That's valuable work they should be rewarded for, so N/A is pretty frustrating just for that reason, even before the fairness question.)
Let me reply separately in the other comment thread here about the general question of the unfairness of N/A resolutions...
My first thought is, can we just declare that we'll follow Metaculus's resolution?
I suppose some traders are aware the creator is gone and are aware mods might follow Metaculus. Most won't be though. So the more involved users are at an advantage.
That's what we want in a prediction market, but we want it regarding to knowledge about the ground truth of the world, not regarding to knowledge about how platform norms will steer who resolves a question. And yes, technically, Manifold's inner workings are part of how reality unfolds, and a good predictor will need to anticipate this, but... I think you get my point.
I'd be interested in how the following market should resolve other than N/A or alternatively: How should a resolution to PROB according to suggestion 5 be determined and who should spend the time and nerves to hear and consider all the arguments for and against, when N/A just very cleanly solves all issues, @dreev:
https://manifold.markets/strutheo/will-chat-gpt-45-release-before-the
@Primer I didn't follow that market but sounds like the true answer is that it was partially released in time -- released to just Pro but not Plus or free users. I do think that's a good case for resolve-to-PROB. (Also traders should've clarified what "released" meant exactly.)
But as for why N/A can be unfair, imagine Trader Alice knows it's going to be tight but has inside information that that Sam Altman is determined to get it released in February. Trader Bob has inside information about a monkey wrench the dev team ran into that's likely to delay it. Maybe Alice knows about the monkey wrench too but believes they'll do what it takes to make the deadline by the skin of their teeth. They ought to be fighting this out with their trades, but now imagine Bob reasons like so: "I'll go all in on NO because if they make it in time I can use arguments like 'it didn't include all users' to say that there's ambiguity and the market should N/A."
In other words, if there's ambiguity it resolves N/A but if OpenAI simply misses the deadline, it resolves NO. Alice has a pretty good case that she was cheated there.
@dreev Yeah, I understand that line of reasoning., but it doesn't apply nearly as strong as you make it seem. Some minor notes follow, then my main point.
Minor notes:
Also traders should've clarified what "released" meant exactly
I suppose you mean there should have been a clarification in the description from the start. Of course, but then a market just resolves normally. We just consider N/A in ambiguous situations.
In your example, Alice could do the same, then Bob would have a case. If Alice is either well-known in the community, or rich, or both, she might reason: "If I buy Yes, traders will follow and this will be well above 90%. Then it will be hard to resolve No. And if traders notice the ambiguities and call for N/A, my fellow Yes traders will argue that, obviously, traders interpreted it as clear Yes, which can be seen from >90% Yes. Also, if the creator would resolve this to PROB, he'll at least subconsciously base his percentage on the last days of trading, which I heavily influenced."
Now my main point:
I think we both really want prediction markets to work, because we consider them the most promising form of information aggregation for decision making. We want Alice and Bob to engage and we want the market to pay out to the one who was right. Ambiguous markets might resolve one way or the other, and depending on your point of view this will be either right or wrong. That's not helping much.
But if Alice and Bob could trade in a market with clear criteria, the one with the better information, the better model of reality will win! If we keep trying to salvage ambiguous markets and keep resolving them one way or the other, those markets will keep attracting traders and liquidity and keep incentivizing creators to create them and keep paying trader bonuses. Consequently, we won't get the market we need, the one where Alice and Bob can fight it out. But if we N/A ambiguous markets, traders will move towards markets with clear criteria, and creators will want to create them.
@Primer I guess I'm suggesting a norm of clarifying before betting and adding the clarifications as they come up. N/A'ing a market for being ill-specified isn't inherently bad. I'd actually like to figure out a litmus test to apply -- maybe we can brainstorm this -- for when N/A is fair vs not. New idea: If any trader speaks up and says, sincerely, that they were betting under the assumption that what ended up happening in the world would be treated as a YES/NO resolution, then assume it would be unfair to resolve N/A.
In section 3 of my writeup here, with the "Will aliens arrive by Christmas?" market, we have the example of someone trying to cheat by pointing out an ambiguity after the fact. The cheater says they didn't think "aliens" meant extraterrestrials and the market should resolve YES. All the sane people say it should resolve NO.
If we can spot the insincerity in that YES argument, great, but let's say we can't. Then by my proposed litmus test, with people arguing seemingly sincerely for the assumptions they'd made (aliens = extraterrestrials vs not), fairness dictates that we hash it out and pick a resolution. If that ends up being resolve-to-PROB (we deem it partially true that "aliens" arrived), that's more fair than N/A, at least in some cases.
So the key would be that people aren't just making post hoc arguments for how to interpret the ambiguous thing that happened but rather are making a convincing case for what they assumed when they made their bets. This might be asking too much. And probably we need more case studies to get better intuition. How might it have played out in the case of the GPT-4.5 release market?
@dreev This might be a pretty good example to get to the crux.
I think it's credible that some traders might think of extraterrestials when they see the "aliens by christmas" market. There were news about crop circles back in the 1960s and Robin Hanson has talked about UFOs recently. But most people's priors for extraterrestials visiting earth within the next few million years are well below 0.01%, and christmas is in a few weeks, and we can't even trade below 1%. Pretty obvious this is about alien immigrants.
What ended up happening in the real world? 176 alien immigrants by christmas, 0 extraterrestials be christmas.
Your suggestion:
Resolve to PROB, maybe 5%, 10%, 50% or 90%, depending on... probably social standing of the traders involved, or amount of Mana they spent?
My suggestion:
N/A.
The market was bad and anyone sincerely interested in betting on alien immigrants or extraterrestial spaceships could have just created a proper market with proper resolution criteria.
Now this
anyone sincerely interested in betting [...] could have just created a proper market with proper resolution criteria.
will not happen if the "aliens by christmas" markets resolve Yes/No, because
The ambiguous market is mispriced and thus
All traders trade in the ambiguous market and thus
Creators even have an incentive to create ambiguous markets (which even are much easier to create)
@Primer I meant aliens-by-christmas as a stylized alternate-universe example where all reasonable people assume aliens=extraterrestrials and go all in on NO because obviously UFOs aren't real. But we pretend that in this alternate universe there's some small chance that UFOs are real. The cheat is to go all in on YES as a free option, to use the finance term. On the off chance that extraterrestrials arrive, you 100x your money. If they don't, you drum up ambiguity concerns and get the market N/A'd.
What's unfair in general is when the chances of settling on N/A depend on what ends up happening in the world.
Back to the AGI markets, suppose I go all in on YES with the following reasoning: If something AGI-like happens, I'll argue that it counts and cash in. If not, I'll point out the ambiguities and argue for N/A. Or on the other side: I go all in on NO and if something AGI-like happens I'll nitpick it and argue for N/A because the nitpicks weren't clarified.
(If those are both equally likely, do they cancel out? I'm genuinely still confused about this.)
Maybe the key is that if you're a NO trader arguing for N/A, the market creator should treat that as kind of morally equivalent to the NO trader arguing for NO. It's all too easy for an argument for N/A to seem to be an appeal to fairness -- "just give everyone their money back as if this ill-specified market never happened, no harm no foul" -- when really it's traders on the wrong side of reality trying to weasel out of paying up.
Candidate heuristic: if any traders are arguing for N/A after the fact then it shouldn't resolve N/A. N/A'ing beforehand is fine. Maybe? But of course beforehand no one's motivated to argue for N/A. (Except you! Nice work being proactive about this in the AGI markets, for real!)
Candidate heuristic: if any traders are arguing for N/A after the fact then it shouldn't resolve N/A. N/A'ing beforehand is fine. Maybe? But of course beforehand no one's motivated to argue for N/A. (Except you! Nice work being proactive about this in the AGI markets, for real!)
Would be a step in the right direction. Note I've been arguing for N/A in cases where I won and in markets where I had no stakes at all, I hope this shows I really velieve N/A to be correct.
I think with your whole argument against N/A, your reasoning focuses on very few outliers. Like someone who spent time marketmaking or some hypothetical cheater betting on longshots and trying to cover his losses. Do we even know of an example for the second case? Or someone like me, who thinks prediction markets are important and actively engages to try to alter the norm around N/A.
But the average users will see a market (more likely if it is heavily traded (for example due to being mispriced (due to being ambiguous))) trade in it and the next thing they know is it resolves against them, because others interpreted it differently. Those will, understandably and rightfully, be disappointed and chances are: might simply leave.
all reasonable people assume
If we're sure we know what everyone (not just the high bankroll traders) assumed: sure. But I posit we just don't know most of the time. Except if we're saying "Well, when I read the question, I assumed X. As I'm reasonable, the only reasonable assumption is X."
I outlined the thinking process of another reasonable person, who comes to the opposite interpretation.