Will I Be Able to Fine-Tune a 1B Parameter LLM on Prediction Market Data to Give Good Outputs by End of February?
10
139
210
resolved Mar 6
Resolved
NO
  • Must have a publicly accessible endpoint (though might be invite only to prevent crashing).

  • Will form resolution council of users to evaluate outputs.

  • This market is subjective, but we have to come up with some kind of scope of input parameters. The wider the scope of possible questions that could be asked, presumably the harder this will be to complete, the narrower the scope, presumably the easier this will be to complete.

End of January Market:

https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par

End of March Market:

https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par-2abd6abdb992

Q&A To Clarify Market (Updating from Comments in Previous Threads):

Q: What's the LLM being fine tuned to do exactly?

A: Answer questions about the likelihood of a basket previously asked questions, e.g. for a question, "Will A Happen?" which resolved as outcome YES, it needs to be able to give a positive answer.

Q: What's the testing methodology?

A: 1. I start with a massive sample set A. 2. I train on part of sample set A, call it B. 3. The output quality will be judged based upon applying it to C. which is B' selected from A.

Q: What are the Limitations? Isn't this all just irrelevant if you spend huge amounts of money on hardware?

A: I'm using an old consumer computer with a GeForce 1070 GPU, so there is a severe hardware capability constraint.

Q: What counts as performance having met the threshold?

A: TBD

Get Ṁ200 play money

🏅 Top traders

#NameTotal profit
1Ṁ21
2Ṁ18
3Ṁ14
4Ṁ13
5Ṁ9
Sort by:

January market resolved NO, put some comments on there as to why and updating my progress:

https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par

Maybe your fine-tuned LLM would be a good fit to participate in this challenge? https://manifold.markets/CDBiddulph/will-there-be-a-manifold-bot-that-m?r=Q0RCaWRkdWxwaA

@CDBiddulph Could you make an OpenLLM version of that contest? Seems like just using GPT-4 (or GPT-5 or better, whatever comes out later this year), will win no matter what. OpenLLM's are a different challenge all together.

@PatrickDelaney If you want, you could make a market for what the score of the highest-scoring bot based on an open-source LLM will be. Just add the tag "Motley Bot Challenge" to your market

@CDBiddulph No thanks, not really my thing to run contests.

@PatrickDelaney Sure, if you managram me 50 mana to cover the cost of the market I'll create it for you

@CDBiddulph I'm just not sure it'll get enough traders to be worth the cost for me personally

@CDBiddulph There, just sent you 100 Mana. Go wild!

@PatrickDelaney Thanks! https://manifold.markets/CDBiddulph/what-will-be-the-highest-score-of-a
I went ahead and spent the other 50 mana on ads for that market.
Good luck with your bot!

More related questions