Must have a publicly accessible endpoint (though might be invite only to prevent crashing).
Will form resolution council of users to evaluate outputs.
This market is subjective, but we have to come up with some kind of scope of input parameters. The wider the scope of possible questions that could be asked, presumably the harder this will be to complete, the narrower the scope, presumably the easier this will be to complete.
End of January Market:
https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par
End of March Market:
https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par-2abd6abdb992
Q&A To Clarify Market (Updating from Comments in Previous Threads):
Q: What's the LLM being fine tuned to do exactly?
A: Answer questions about the likelihood of a basket previously asked questions, e.g. for a question, "Will A Happen?" which resolved as outcome YES, it needs to be able to give a positive answer.
Q: What's the testing methodology?
A: 1. I start with a massive sample set A. 2. I train on part of sample set A, call it B. 3. The output quality will be judged based upon applying it to C. which is B' selected from A.
Q: What are the Limitations? Isn't this all just irrelevant if you spend huge amounts of money on hardware?
A: I'm using an old consumer computer with a GeForce 1070 GPU, so there is a severe hardware capability constraint.
Q: What counts as performance having met the threshold?
A: TBD
1,000
3.00January market resolved NO, put some comments on there as to why and updating my progress:
https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par
Updated notebook on the topic... https://github.com/pwdel/gpu-jupyter-tensorflow/blob/main/volumebindmount/llm_training_experiments/1B_QLORA_TRAINING.ipynb ... 44 days left
Maybe your fine-tuned LLM would be a good fit to participate in this challenge? https://manifold.markets/CDBiddulph/will-there-be-a-manifold-bot-that-m?r=Q0RCaWRkdWxwaA
@CDBiddulph Could you make an OpenLLM version of that contest? Seems like just using GPT-4 (or GPT-5 or better, whatever comes out later this year), will win no matter what. OpenLLM's are a different challenge all together.
@PatrickDelaney If you want, you could make a market for what the score of the highest-scoring bot based on an open-source LLM will be. Just add the tag "Motley Bot Challenge" to your market
@PatrickDelaney Sure, if you managram me 50 mana to cover the cost of the market I'll create it for you
@PatrickDelaney Thanks! https://manifold.markets/CDBiddulph/what-will-be-the-highest-score-of-a
I went ahead and spent the other 50 mana on ads for that market.
Good luck with your bot!