https://github.com/KingSupernova31/manifold-search/blob/main/maintainMarketData.js
When I run this script on my VPS, it runs fine for the first few hours, but eventually it sends the machine into a state where it's unable to process requests, and all other websites hosted on that machine become unreachable.
M$500 to the first person to explain why this is happening and provide a fix that works.
1,000People are also trading
The fact that the server dies is an argument against external rate limiting, unless the way the script is set up makes request failures happen in a worse way
what do the machine's ram, CPU and disk space usage look like as time goes by as it's running? what does ps or other system onitoring commands output?
Is the machine pingable when it's in "bad" state?
How do you get it back to normal once it enters bad state?
This could be the reason: https://docs.manifold.markets/api#usage-guidelines
Feel free to use the API for any purpose you'd like. We ask that you:
Keep your bets to less than 10 per minute, amortized (transient spikes of over 10/min are okay).
Keep your reads to less than 100 per second.
Your script looks extremely read-intensive... you are probably being rate limited
Ok something similar has happened. It's not blocking other requests to the website, but the update script has itself become largely unresponsive. Opening up the logs shows that I'm getting a bunch of connection errors. My best guess is that some of the requests I'm making end up hanging forever, and it eventually clogs something up so that future requests start failing. I'm going to try adding a timeout to every request to see if that fixes it.
I don't have a smoking gun for you, but there's two things that I'd check.
The 200 ms recently changed markets loop looks a bit excessive. Dunno if your VPS provider has any IOPS limits to run against.
The array-as-queue with splices and (un)shifts looks like a good candidate for accidental quadratic complexity. Heap queue with two priority levels would be easy to do.
Assuming Linux.
Check memory/swap usage over time. Extreme slowness and processes dying is a common symptom of running out of memory. This can take minutes to hours to resolve on a bad day. Another indication of this would be OOM killer invocations in your systemd journal or dmesg output. Or maybe you can just measure the memory use of a single invocation of the script using one of various tools.
If this is the case, a band-aid solution is to install
oomd(https://github.com/facebookincubator/oomd/) or limit the memory usage of the process using cgroups, e.g. with running it under a systemd service with configured limits or maybe firejail.Longer-term, you might want to revisit loading all market data all at once from a big json file. Maybe throw it in a sqlite database or stream it one value at a time so that the whole file is never put into memory. Maybe you want JSONL here?
The pm2 comment at the top also makes me thing this is likely after reading the pm2 docs at https://pm2.keymetrics.io/docs/usage/memory-limit/ that say the memory limit param relies on polling every 30s. Memory can easily fill up faster than that.