It looks like bug causing most (but not all) of the profit spikes was fixed. I'm going to resolve this 90%.

@Yev Hmmm… mostly fixed doesn’t seem to satisfy the stated resolution criteria

@Yev I highly disagree with that resolution.

@Yev I still see spikes, how is this "mostly" fixed?

Manifold made a change that greatly reduces the number of spikes - see James's comment below. FWIW I looked at my profit chart for the first week this market was open (Nov 27 - Dec 3) and see what looks like 6 spikes, while in the last week of Feb I see 1 spike.

@jack Right, but the market was about whether it would be fixed, not about whether it would be worked on or about whether it would be improved.

@jack It looks like the root cause is still there, the new process is just better at masking it. As the number of open markets and the number of bets go up, the spikes will become more common again.

@Odoacre One of the major root causes was fixed: "we fixed one problem by fetching the user balance at the same time as we fetch all your bets, so the ability for a race condition to occur is much less than it used to be." solves the problem that people speculated was a cause earlier: "Possibly it was failing to sync up the addition of new shares with the mana leaving the account."

I think either interpretation of this question is understandable:

Resolves YES if whatever is causing them has been fixed by end of February 2023.

They were not all fixed, but it is definitely true that some of the causes of profit spikes were fixed. I feel that a percentage resolution is reasonable, and that a NO resolution could also be reasonable.

I don't have any stake in anymore in this but I feel this should resolve No based on this.

Don't have time to investigate, but my impression was similar to @jack where the old buggy spikes were unbelievably narrow

@PatMyron this is a 30 min spike. exactly the same as all my other previous profit spikes

@PatMyron i currently am in a 43k or so profit spike. what now?

@MarcusAbramovitch we both hope it doesn't go back down 😅🤞

@PatMyron haha, i fucking wish i made that. sadly it's already gone

Most spikes were 15 or 30 min IIRC, so this matches.

@jack

Maybe @JamesGrugett could weigh in since the discord message mentioned him working on getting rid of the spikes

@PatMyron We've switched to a process that updates 500 users every 1 minute, or about 40 minutes for all users.

Regarding the spikes, we fixed one problem by fetching the user balance at the same time as we fetch all your bets, so the ability for a race condition to occur is much less than it used to be.

I wouldn't be surprised if there are still cases where it would spike though. The best solution would be a verifiable log so we never have inaccurate profit data.

@JamesGrugett might that explain why some of us remember the spikes being narrower previously? As more users use manifold, the time to process updates for each individual user would increase if the process is fixed at 500 users/min

This is on Jack's portfoliio and seems to be the most egregious example I could find.

These seem to be all the mades Jack made around that time, none of them can account for the 8k profit spike.
I don't know if Jack had a lot of shares in a market that moved a lot during that time, but of his open markets only https://manifold.markets/memestiny/dan-stock-permanent has more than 2k profit and that only had a 3% change on his 4500 shares so not enough.

Hopefully I didn't miss anything.

EDIT: Jack also made trades on these 3 markets just before the spike:

But these were all just buying from ~95% to 99% and so only made at most a couple hundred mana in profit.

@Bot Maybe this market caused the spike https://manifold.markets/jack/profit-share?r=Qm90, though I don't see why it would be a valid spike.

@Bot I think the profit going up looks legit - I had many big market moves right before the profit went up on the charity markets like https://manifold.markets/SG/will-manifold-have-100k-in-donation and https://manifold.markets/Duncan/will-manifold-for-charity-reach-100 and https://manifold.markets/egroj/will-i-donate-more-than-1100-via-ma - I had a lot of shares and I bought more and moved the price up, that was at exactly the right time for the profit spike and I think that adds up to a few thousand mana of profit.

I don't know why the profit went back down though.

It also doesn't look quite like the bug spikes to me - this one lasts 39 minutes which is a bit longer than the spikes were, I think.

Oh actually you're right, at that time the price only went from ~95% to ~99%, I misread the timestamps. Not sure where the profit is coming from then.

@jack It seems like the minimum interval between updates is 39 minutes.

This bug seems to be fixed. I will resolve this market YES tomorrow unless someone comments convincing counter-evidence.

@Yev they couldn't have waited one more day 😭

@Yev It might be that new instances of the bug are not occurring in the past few days (I can't confirm that one way or the other), if so, that's one half of the puzzle solved.

But that still doesn't fix "whatever is causing [the mysterious profit spikes]" in historical data, which is the other half of the puzzle. Presumably the thing causing the spikes in the historical data is having bad historical data as a result of some bug. Unless there's a massive backfill to adjust a bunch of timestamps and rewrite the historical data, I think these mysterious historical spikes are unlikely to ever be fixed.

@JimHays Right, the graphs are computed and then stored, so they will still show the buggy spikes. I interpreted the question as just requiring a fix forward, not requiring historical data to also be fixed.