Must have a publicly accessible endpoint (though might be invite only to prevent crashing).
Will form resolution council of users to evaluate outputs.
This market is subjective, but we have to come up with some kind of scope of input parameters. The wider the scope of possible questions that could be asked, presumably the harder this will be to complete, the narrower the scope, presumably the easier this will be to complete.
End of February Market:
https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par-1e2a645ec277
Q&A To Clarify Market (Updating from Comments Below):
Q: What's the LLM being fine tuned to do exactly?
A: Answer questions about the likelihood of a basket previously asked questions, e.g. for a question, "Will A Happen?" which resolved as outcome YES, it needs to be able to give a positive answer.
Q: What's the testing methodology?
A: 1. I start with a massive sample set A. 2. I train on part of sample set A, call it B. 3. The output quality will be judged based upon applying it to C. which is B' selected from A.
Q: What are the Limitations? Isn't this all just irrelevant if you spend huge amounts of money on hardware?
A: I'm using an old consumer computer with a GeForce 1070 GPU, so there is a severe hardware capability constraint.
Q: What counts as performance having met the threshold?
A: TBD
1,000
3.00I wanted to provide an explanation even though it seems pretty much everyone bet against me, which is reasonable. Basically I have been working with a much more powerful equipment at work, and concentrated on tooling. So while I have been able to figure out how to do quantized training, I haven't had the time to develop any of this on my personal equipment and repo.
My hope is to be able to translate some of what I have done on my smaller, older machine. Goal for next month:
https://manifold.markets/PatrickDelaney/will-i-be-able-to-finetune-a-1b-par-1e2a645ec277
Updated notebook on the topic... https://github.com/pwdel/gpu-jupyter-tensorflow/blob/main/volumebindmount/llm_training_experiments/1B_QLORA_TRAINING.ipynb
14 days left
@vluzko Good question! Here's an idea, please let me know what you think. 1. I start with a massive sample set A. 2. I train on part of sample set A, call it B. 3. The output quality will be judged based upon applying it to C. which is B' selected from A. 4. The natural constraint will be, I'm using an old consumer computer with a GeForce 1070 GPU.
@vluzko We might derive further performance criteria ideas from this market, though the purpose of my market is not to create a bot, but rather just fine-tune the LLM such that it could hypothetically be turned into a bot by someone else. https://manifold.markets/CDBiddulph/will-there-be-a-manifold-bot-that-m?r=Q0RCaWRkdWxwaA
@jskf Answer questions about the likelihood of a basket previously asked questions, e.g. for a question, "Will A Happen?" which resolved as outcome YES, it needs to be able to give a positive answer.