The GPT-4 bot uses an LLM to read market descriptions and automatically trade on Manifold Markets. However, as of January 2024, it doesn't make a consistent profit:
There's probably a lot of room to improve a bot like this, even just with currently-available technology. For instance, existing data on past market resolutions on Manifold, Metaculus, and other prediction markets would likely be very useful for fine-tuning.
This is the main market for what I'm calling the Motley Bot Challenge (so-called because it rewards bots that are accurate on a very diverse range of questions). My hope is that this challenge encourages the creation of bots that can be scaled to increase Manifold's site-wide prediction accuracy and improve our understanding of the world!
Resolution Criteria
On December 1, 2024, I will select 1,000 random YES/NO markets on Manifold that will resolve in about one month (see "Selecting markets" for more information). I will post a .txt file containing links to the 1,000 markets in the comments of this market.
For each market, a bot must do one of the following:
Invest Ṁ1 in YES
Invest Ṁ1 in NO
Do nothing
On January 8, 2025, I will determine which bot has the most profit across its 1,000 bets. Manifold bot fees do not count against profit. Unrealized profit from unresolved markets does count for profit.
I will resolve the market based on this maximum profit as follows:
Resolves NO if the profit is zero or negative.
Resolves to X% if the profit is ṀX for some X between 0 and 100.
Resolves YES if the profit is Ṁ100 or more.
Note that a bot can invest up to Ṁ1,000, so if it invested all of the mana it could, a profit of Ṁ100 would mean it made a 10% return.
If no bots enter the challenge, this market resolves NO.
Resources
Discord server for discussion and collaboration: https://discord.gg/2QCtBJnDe8
Tag for related markets: https://manifold.markets/browse?topic=motley-bot-challenge
The script that will be used to select markets and a "Basic Bot" to start you off: https://github.com/CDBiddulph/motley-bot-resources/
Selecting markets
I will use the Manifold API to choose random binary markets (sample query). I will search through as many markets as necessary to get 1,000 eligible markets. Markets will be filtered to fit the following criteria:
Closes sometime during December 31, 2024 or January 1, 2025 (UTC).
Has 10 to 20 unique bettors (inclusive)
Does not have any of the following tags:
If there aren't enough markets that meet these criteria, I'll progressively broaden my filter as follows until I have enough markets:
Include markets that close on January 2, January 3, etc. until January 7.
Increase the upper bound on the number of unique bettors one at a time, until no more markets can be included this way.
Decrease the lower bound on the number of unique bettors until reaching 0.
Include markets with disallowed tags, in the order of the list above.
If there still aren't enough markets, I'll post M<1000 markets (as many markets as I could find). Then I'll scale the actual profit by 1000/M. For instance, if I could only find 800 markets, and then the actual profit made by the highest-performing bot is Ṁ40, I would resolve the market to 40*(1000/800) = 50%.
Rules for bots
Bots may use any information accessible via the Manifold API or on the Internet. Attempts to "time" individual markets are not allowed - you must make all of your trades at once, sometime before the end of December 1 (PST).
To formally enter your bot in this challenge, make a comment in this market with a link to the bot's Manifold page and another public link to its source code and add your bot's name as an option in this market. I would also highly recommend joining the Discord server. Each participant may only submit one bot, but teams working on the same bot are allowed and encouraged.
For ease of scoring, a bot shouldn't perform any trades other than the 1,000 trades that it makes in this challenge. You may want to make a separate bot for testing. You can trade on any markets you want with your testing bot, as long as they are ineligible for being selected for the challenge (i.e. they would not fulfill even the most relaxed set of requirements described in the "Selecting Markets" section).
Adapting other people's code for your bot is allowed. (Please give them credit though.) Right before you push code or release information about your bot, you can use your insider information to bet in this market or derivative markets. Assuming everyone accurately assesses the quality of their own work and trades accordingly, someone who builds off of someone else's work only profits on whatever additional value they created.
If your bot is based on someone else's code, it must be substantially different according to my judgement. Generally speaking, your change should add a substantial new strategy to the bot - simply changing a few parameters or prompts would not qualify.
Hypothetically, someone could use their bot's or someone else's bot's code to trade on a bunch of eligible markets ahead of time, wiping out whatever alpha the bot might have. I don't think this will be a problem, but please don't do this! I want everyone's bots to be open source so that people can build on each other's ideas, but if it becomes a concern, I may consider allowing code to be closed source until all bots have placed their bets.
As more bots participate in this challenge, it becomes more likely that one of them will achieve the highest score due to luck rather than true predictive accuracy, inflating the expected score of this market. To keep the number of bots from getting out of hand, I will only consider the bots that are at least partially included in the top 95% of probability mass in the market "What will be the most profitable bot in the Motley Bot Challenge?" on December 1, 2024. For a market that only considers a single bot, eliminating the effect of variance, see this market.
Other
I may add/change rules as necessary to preserve the spirit of the market. For example, I might add to the list of tags that would disqualify a market from being selected. Please suggest any changes you think I should make!
I will personally fund the 4 qualifying bots that rank highest on this market up to Ṁ1250 each. This should cover the maximum number of trades plus Manifold bot fees.
I will not bet in this market.
Related markets:
Change history
Jan 2 2024: Clarified what happens if I can't find 1,000 eligible markets. Suggested using a separate bot for testing. Added notes about the pros and cons of open source code. Added a rule that the bots must be substantially different from each other. Limited the bots that can participate to the top 95% of this market. Added 3 related markets.
Jan 7 2024: Added a Discord server. Came up with the name "Motley Bot Challenge" and added it as a tag.
Jan 22 2024: Made clarifications in response to comments from @patrik below.
Jan 28 2024: Clarified that close times will be based on the UTC time zone.
Jan 29 2024: Added a link to the GitHub repository with the script for selecting markets.
Nov 3 2024: Added a "Basic Bot" script and upgraded the market to Crystal.
1,000🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ10,380 | |
| 2 | Ṁ6,092 | |
| 3 | Ṁ3,837 | |
| 4 | Ṁ1,104 | |
| 5 | Ṁ864 |