I claim that resolving N/A is bad, that it's usually worth the effort to untangle ambiguities and hash out what's fairest. That can be exhausting, invidious work and it's understandable if you're not up for it, but at least consider it a last resort to resolve N/A. And if traders are lobbying for an N/A resolution, that might be all the more reason not to do it!
But this is easier said than done. Sometimes you pose an innocent-seeming question and traders make different assumptions and bet accordingly. Later, the unanticipated ambiguities come to light and all possible resolutions feel grievously unfair to one set of traders or another. And it's possible that N/A is indeed what's least unfair. But here's a collection of reasons to resist it:
1. It's anticlimactic and throws away people's work
This one seems obvious and relatively minor. Resolving N/A unwinds everyone's trades as if they never happened. Less obviously, that includes clawing back profits of those who correctly predicted a price movement and then cashed out. This invariably surprises people.
2. It spoils incentives
This is kind of a corollary to reason #1 but is also a counterargument to a pro-N/A argument. Namely, that the real value of a prediction market is to see the market probability evolve. The final resolution, or lack thereof, doesn't undo that value. That's true but resolving N/A sets a bad precedent, making people less motivated to do the work to make market probabilities accurate in the first place if they think N/A is at all likely.
3. It can be a way to cheat
Here's an extreme example to illustrate how bad this can be. Imagine creating a "Will aliens arrive by Christmas?" market. You dump all your money on YES. If aliens show up, resolve to YES, win big. If aliens don't show up, you start stirring up fairness concerns. It wasn't specified whether you meant extraterrestrial aliens. Etc. Get the debate going and then suggest that maybe the fairest thing is to resolve N/A. Everyone gets their money back, no harm no foul. But it was totally rigged! You made a big bet on YES and then found a way to weasel out of paying up when you were proven wrong.
It's not normally so blatant but it's worth having that in mind. "Just give everyone their money back, no harm no foul" can be extremely incorrect. Totally harm, totally a foul. Be very skeptical of anyone suggesting that resolving N/A is what's fairest.
Rule of thumb: if the ambiguities that arose were no more likely to arise for any of the ways the underlying question might've played out, then N/A may be what's least unfair. It can be a judgment call.
4. It can distort market probabilities in decision markets
Say you want to know whether, if you buy Apple's VR headset, you'll actually use it enough that you'll be happy you bought it. So you make a market asking exactly that. You might naturally commit to resolving N/A if the market convinces you you'd regret the purchase. No purchase, no way to resolve the market. The problem is that this can spoil the incentives for traders and leave the market quite biased towards YES.
Imagine I'm pretty sure you'd regret getting the VR headset. If I drive the price to my true estimate of, say, 10% that you'll like it, my upside is cut off. If I'm wrong and the price goes back up and you buy the thing and have no regrets, I lose money. If I'm right then you defer to the wisdom of the market, don't buy the thing, and resolve N/A. I'd be paying up if I'm wrong and getting no payout if I'm right. Thus I won't drive the price down enough to convince you not to do the thing. I might not bet NO at all, in case others drive the price down that far.
What should you do instead? Maybe resolve to the market price in the case that you don't buy the headset. That does risk market manipulation. Driving the price to 1% is guaranteed profit if I think I have enough money to keep the price there. At that price I know you won't buy the headset and thus you'll resolve to the market's estimate of a 1% chance of headset happiness. Ka-ching. But my own meta-prediction is that that's too risky to try to exploit in practice. Especially if you don't commit to a particular threshold for doing vs not doing the thing. Any would-be manipulator has to worry that you might buy the headset despite the market seeming 99% sure you'll regret it. If you do, and don't in fact regret it, the manipulators lose their shirts.
5. Ambiguous outcomes are often better resolved to a probability
Did the thing being predicted kinda/partially/ambiguously happen? You don't have to endlessly debate YES vs NO and give up and resolve N/A when neither side can claim a decisive victory. Instead the creator can make a judgment call like "this counts as 75% YES" and just resolve to that. Resolve-to-PROB, it's called. It may seem weird but is pretty exquisitely fair if that 75% is the best reflection of the weird reality that ended up happening.
Exceptions
A. Questions that are fundamentally nonpredictive, like "Is Trump good?".
B. When the resolution criteria explicitly stipulate an N/A resolution if such-and-such happens. Like "Will Biden beat Trump? Resolves N/A if either of them dies before the election."
C. When an error was made in specifying the resolution criteria -- such as changing the criteria after some people already bet -- such that any possible resolution would be truly unfair to some subset of traders.
(I originally said some of this in a "will DALL-E be able to draw blue grass and a green sky?" market and a "will I regret bleaching my hair?" market. It was in the latter that user @Conflux pointed out the potential for market distortion in a "will I regret XYZ?" market. For more on trickinesses with decision markets, see Prediction Market Does Not Imply Causation.)
1,000
3.00reposting for reach: NA poll with a thoughtful writeup from @dreev on reasons it's best to resist the option
There's another behaviour that the NA option can facilitate, which is a variation of #3
Let's say that someone makes a market of medium complexity, and doesn't define the resolution conditions particularly well. But they certainly know what they intend to do for the majority foreseeable cases.
People bet and the market swings in a certain direction. Then a new bettor comes along, and realises there is an ambiguity. Instead of asking a clarifying question, they just bet the higher reward position, having already decided that if the creator 'rules' against them they will demand an N/A.
I think there is a general problem on Manifold where if a question seems poorly defined, and your good faith interpretation of what is asking suggests it is very incorrectly priced, the sensible thing to do is to ask the creator what the intention was, because it looks like you might be wrong.
However, if you ask, and the creator says "No no, your interpretation is correct", the chance that you will be the first one to see that information is probably not very high, and then someone else benefits from your insight and takes all your EV.
@JoshuaWilkes my solution is to make an initial bet on both the question and the probability that my interpretation is correct, and just take the hit like a manifolder when I'm wrong
Beautifully said, and all the more reason that if a trader is "demanding" N/A, that's a strong signal that it shouldn't resolve N/A. And I strongly agree that the Manifold norms are caveat emptor. Ask clarifying questions before trading!
Arguing ex post facto that the resolution criteria were ambiguous is no good.
/agentydragon/will-google-search-include-a-chatbo
I don't recall the exact details, there was a wording ambiguity involving by/at. Either YES or NO might have been correct. The question was still live and relevant. In situations like this, I generally expect people to ask for N/A but not get it.
/HelenD/will-miami-marlins-beat-new-york-me
The game in question was postponed due to weather and played the following day. Neither YES or NO seem correct. Circumstances had made the question irrelevant and moot. I generally expect N/A in these cases.
I think of mootness as a key factor in whether N/A will be used, but every situation is different.
PS: I rearranged everything in the market description to be just a list of reasons not to resolve N/A, with the arguments in favor discussed along the way.
I'm still thinking about how to articulate a generalization of reasons 3 and 4. Something about needing the probability of N/A to be the same in all possible worlds, lest it be unfair and/or distortionary.
Say you want to know whether you'll regret it if you buy Apple's VR headset. So you make a market asking exactly that. You might naturally commit to resolving N/A if the market convinces you you'd regret the purchase. No purchase, no way to resolve the market.
In this case you might make a pair of markets, "Will I regret purchasing Apple's AR* headset" and "Will I regret not purchasing Apple's AR headset". You make a decision based on the spread between them and N/A the counterfactual after you choose. Either way, I think N/A is essential for conditional markets, and that's the only way I intentionally use it.
@Sailfish Hmm, but is it possible to regret not buying it? As soon as you started to have such regret, you'd go ahead and buy it. I'm not sure the additional market is accomplishing anything in this case.
Related to decision markets: https://manifold.markets/dreev/how-many-instances-of-prognootling
(Btw, I flipped the example in the description to be predicting happiness rather than regret. I kept confusing myself with the double negatives there.)
but is it possible to regret not buying it? As soon as you started to have such regret, you'd go ahead and buy it.
Then the regret market would resolve "Yes" and the condition for the "Regret purchasing" market would be triggered. I assume there's a resolution time. For instance, I say I want to know if I should purchase the headset in seven days. I make the pair of markets, and they both resolve in say two months. "Regret buying" is at 70%, "Regret not buying" is at 30%, and I decide not to buy it. Two weeks pass, and I feel like I do actually want to buy it, maybe I try it out somewhere and like it a lot or something. If I do end up buying it, say, a month later, I would have liked to have this information earlier and so I "Regret not buying" it. I agree the wording is kind of weird, at least, it's not a common way to use "Regret" but I think it's correct.
Maybe it's more intuitive to think that never buying anything isn't some secret strategy to optimize happiness. There are some cases where you'll think that you actually did want something, and would prefer to be in the counterfactual world where you did in fact buy it at some previous time.
I guess the key in my VR headset example is that we're more likely to get an N/A resolution in the universe where the VR headset sucks. The headset sucks => the market knows that => you listen to the market and don't buy it => you resolve N/A.
Similarly for the "will aliens land by Christmas?" scam. The market is more likely to resolve N/A in the universe with no aliens.
So the general principle may be that if you resolve N/A you should be certain that your probability of doing so is consistent across all universes that the market is predicting among. Something like that?
Easier said than done though. Counterfactual reasoning can be confusing.
If I have to vote, I will vote for "preaching to the choir", but, in defense of N/A:
On some markets I create, I know there is a low-odds thing that could reasonably happen (more often than 1 in 10,000 but less than 1 in 20), but I do not expect it to happen. Whether or not the low-odds thing happens is not valuable to me and generally not related to the important part of the question I am asking. Due to the structure of the market, it is difficult to price in the low-odds thing, so I will sometimes explicitly declare what the market will do if the low-odds thing happens, and it is often N/A.
Especially in cases where the odds of the low-odds thing are hard to determine, I'm usually much, much more interested in A vs B, than A vs B vs Low-odds-thing. The multiple choice question format sometimes allows explicitly factoring this in, but in my experience, users have a very difficult time pricing low-odds-thing because the site does not seem to want us to be able to price low odds events.
---
For cases where unforeseen events create ambiguities that could result in N/A, I think it is important to make sure traders understand they are trading in a situation of extreme ambiguity. If I know I'm trading on thin ice, I will have a lot less reason to feel betrayed if the market does not turn out the way I think it should. I think it would be good for the platform to support users by allowing creators, moderators, or users to post a warning flag so users can review the situation carefully before proceeding to bet on markets that have reached this state.
@Eliza Ah, this sounds like an important caveat that may be worth incorporating into my little essay here. If you pre-commit to a specific condition under which you'll resolve N/A and that won't bias the market probability (see the description here for an example of when it does! I have to think about how to characterize in fully generality when it does or doesn't) then resolving N/A is fine.
@TheoSpears Interesting! I haven't thought this through yet; are you thinking that that mitigates some of the CONs here? It at least offers traders a way to hedge. Maybe you didn't mean this as a serious solution, and my guess is it doesn't help too much. But it can't hurt!
@dreev It seems like a possible strategy some traders could use to mitigate the impact of an N/A resolution if it seems likely.
N/A is totally necessary for learners, therefore for growing Manifold into anything sustainable. If the consequences of structuring a market suboptimally are punitive, hardly anyone will stick with it and improve. How to balance that with discouraging trolls and unconstructive behavior I don't know but it can't be impossible.
ETA: Could it connect to the ratings system somehow? Maybe only N/A resolutions get rated. Ratings not super meaningful as is.
@ClubmasterTransparent Fair. I think my PRO #1 and maybe #2 cover this. It's very possible for an N/A resolution to be the least bad option. In any case, I wrote this up in part so that, as newbies get up to speed, we have something to point them to to make the case that it's worth some work to avoid N/A resolutions.
@ClubmasterTransparent Can you give an example? I don't think it's 'totally necessary' in a literal sense, maybe a factor that matters on balance but idk. Do you mean it's important for new market creators or new users betting on markets that n/a?
@jacksonpolack @dreev thanks for thoughtful responses. I meant new market creators. To realize any of our various great ambitions for Manifold, it needs to attract a wider cross-section of good users. I have learned a lot about AI here, among other things, but I almost got manic when my Sam Altman market blew up on my first active week here, just trying not to f up. Example of N/A that people liked resolution and I learned from
I agree that especially for large markets, N/A should be used sparingly. Although it can be the fairest option in some cases, it's unfair to resolve a market N/A just because of some technicality or small ambiguity if it's clear which answer has the much stronger case. After all, the people who correctly predicted the event should be rewarded, even if there's some slight ambiguity to what they were predicting. It would be unfair to withhold the reward unless it's very uncertain whether the prediction they made actually included what came to pass.
I actually recently had a case where this became relevant in my ChatGPT market. The original description didn't specify what would happen if there was an error message, so I clarified that I would count this as the same thing as "refusing to choose an answer", which counted as an "Other" resolution. But when the possibility was first brought up, I briefly contemplated saying that the market would resolve N/A in that case... until I immediately dismissed it as a terrible idea, in part because it looked like that was what would actually happen (it did), and I didn't want it to resolve N/A. So instead, I picked what seemed to me like the option that had obviously the best case to be made for it - lumping it in with all the other hypothetical possibilities where ChatGPT doesn't give an answer.
I agree. I think there's an exception though, if a market has like 4 traders and no volume and it'd take a lot of effort to determine how to resolve it it's fine to N/A it and managram people their potential winnings.