“If the code turns out to be correct, the market will resolve NO.”

misresolved? ha just kidding. Great data-detective work.

@deagol the code was not correct with respect to the goal of calculating the average of The Market, it needed to be modified to filter redemptions out.

Went through this with @Gabrielle. The theory is that the problem is how you are treating limit orders. Only some of the limit order fills actually move the market.

(For example if market is at 60% and I put in an order for NO at 70% the market will not move, but if I put in an order for NO at 50% it will move the market to 50%)

here's some code that shows the issue (again, based on @Gabrielle 's work)

https://pastebin.com/1Ti9FT7k

this prints:

current average 0.3645699493261204

which matches the result using data from the console

@Odoacre I don't think so, the problem is redemptions only. Entirely unfilled limit orders are present in the data, but they have probAfter set to the same as whatever the market was already at, so they do not affect the average probability calculation.

In your modified script you are excluding redemptions and that is doing all the work, your exclusion of entirely unfilled limit orders doesn't change the result.

@chrisjbillington oh, I see. My code filters out redemptions because they have no fill object (I assumed they were payouts of some kind).

Anyway, seems like we have reached a conclusion, happy days!

Haven't checked the code and I assume this isn't relevant, but I figured I'd mention that filled limit orders can also result in an unmoved probability. (Someone buys into a limit order without filling it fully, someone else later buys more into it, still without filling it.)

@IsaacKing that's right, the code will correctly filter those fills out. It does not really matter though.

@IsaacKing right, but since those bets still have probAfter set to the actual market probability at that time, the calculated average still comes out correct. So they can be filtered out, but the result won't be wrong if they're not.

The data from the API contains additional bets that are not in the data sent to the browser (highlighted diff of timestamps and rounded probabilities after each bet shown here):

Some of these bets have the same probAfter as the bets that precede them, and so their presence does nothing. But some have different probabilities to the bets around them, and do affect the calculation.

I can see that if you ignore these additional bets, the probBefore and probAfter of all other bets form a continuous probability history (the probBefore of each bet is equal to the probAfter of the bet preceding it). Whereas the additional bets have an equal probBefore and probAfter that don't necessarily match up with preceding or following bets.

These additional bets have isRedemption=True, other bets have isRedemption=False.

I don't know what a redemption bet is, but I see this bug report in the discord:

Whatever they are, they're not supposed to show up. So they're present in the API data, but not intended for display on the page, and indeed, we see they are absent from the data as scraped from the browser.

If I filter the bets obtained via the API to not include these redemption bets:

bets = [bet for bet in bets if not bet.isRedemption]

Then I get 36.457, in agreement with @AlexbGoode's calculation of the average probability for The Market, and my own using browser-scraped data.

@chrisjbillington Those are the share cancellations from when people buy shares of an opposing type.

@chrisjbillington Very thorough investigation. My guess is that a redemption is when you have both YES and NO shares and they are redeemed for 1 mana. These influence the market in a way that, to me, seem not so intuitive.

A related market where redemption and limit orders had a weird interaction:
https://manifold.markets/FlorisvanDoorn/will-this-markets-value-ever-hit-20-ff4a3e6408a0

@IsaacKing Hm. Makes sense. Though I would have expected that these bets would move the market and have meaningful probBefore and probAfter values.

wait so you all saying my highly speculative layman’s intuition that there were probably bogus transactions somehow being counted was not that off? like I clearly remember my very first response to Chris on this yesterday, where I specifically mention the YES+NO share cancellations. However, calling these spurious data “my ball’s hairs” did cross a line, I fully admit. ;)

Thanks again, Chris, Alex, and all. This was as exciting as it gets, down to just single-digits in basis points! (1/10000)

@chrisjbillington the bets are recorde, the extra stuff I assume is the redemptions, where 1Y+1N=1 mana redeemed

*recorded

@chrisjbillington “bets = [bet for bet in bets if not bet.isRedemption]”

this is code poetry 🙌🏼

From my comment at "the market":

The data in the spreadsheet is the data manifold.markets sends to my browser periodically. My educated guess it that this data is used to update the graph we see on this site. The naming of the variables supports my guess. The data is called "points.x" and "points.y", which is a common way of naming plot axis. But I have not (yet?) reverse engineered the site nor have I looked closely at the JS code.
I have also not looked at the official manifold API (why do that if the data is already there?).
The data from the API could be different, why I do not not know. Only manifold will be able to tell us.

@AlexbGoode The data you get from the API seems to be the bets made. Manifold seems to process this data internally before sending it out the be displayed.

To summarize:
Your script is not wrong. The data is different. My guess is that Manifold processes the data to display in some way. Why Manifold does that is, imo, outside the scope of this question.

From a quick look, I can't see any obvious mistakes in your code, IMHO the discrepancy is becasue you are using a different source of data. If I understand it right @AlexbGoode is using data collected from the browser console, you are using data from the API. Not sure what @Gabrielle is using yet, I have reached out on discord, but it will take some time to figure this out, since we are in different time zones.

Do you mind extending the end time for this ? I will definitely have a look, but might be busy in the next few days

@Odoacre sure. Extending to a week, and reserve the right to extend further. Apologies to anyone who was banking on quick resolution, can reimburse if there are gard feelings.