Resolves NO.

all this requires is memory right? Let's imagine gpt-4 was 1TB of weights, which means ~2Tflop/token.
The Mac M1 integrated GPU can do like 10Tflop/s, so that's 300 tokens/minute only considering flops.
The Mac M1 has 400GB/s of memory bandwidth, which limits it to 20 tokens/minute.
So all you need is 1TB of memory, which would cost $2400 today.

If you used an NVME ssd instead of ram, you only have 5GB/s bandwidth, so only 3 tokens/minute, which only costs $50!

So seems guaranteed that you can get completions within a day, even on a $500 budget, let alone $3k

@TaoLin It also requires you to have access to a GPT-4-equivalent model's weights.

@osmarks the top post doesnt require it to be open source, so if someone chooses to reveal that they can run gpt4/gemini/whatever closed source model on their PC, which they obviously can, that might count

I have llama2 7B running on a single GPU with similar results as https://github.com/ggerganov/llama.cpp
So, few tokens per second. I have no doubts the full llama2 70B can run on a single consumer machine to generate 500+ tokens in less than 24h.

@GiovanniRizzi LLaMA 2 is not at all GPT-4-equivalent.

150elo per 10x compute.

GPT-4 300 elo ahead

$100m dense, or ~$3mm if everything were done state of the art. And linearly stacks.

You've got Yan LeCun pointing to this paper, claiming GPT4/Bard level performance for LLaMA 65B. That's a fairly god argument for being able to achieve this toward the end of the year because I believe you can already run LLaMA 65B on a tower server that costs less than $3k USD. https://arxiv.org/abs/2305.11206

LIMA: Less Is More for Alignment

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two…

Not implausible (with an enormous cache, incredibly sparse moe, and single digit millions to train)

Just such a brutal architecture to deploy at 0.0X tokens per s that can’t imagine why it would be attempted

All leaderboards board to 2-3yr away

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (May 9)

As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy.

Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

@fleventy This isn't relevant. It's just dispatching efficiently to LLM APIs.

@osmarks something like this could be considered an ensemble model (one that has sub-models).

Depends on the definition of model. Single ANNs only, or can it be more sophisticated? @LarsDoucet

@fleventy That does blur the line somewhat but I think it's outside the scope of what I was intending.

100TB of weights and 10PB of cache

Notice that GPT-4 can quote back entire copyrighted books and start buying tape drives

Assuming it's even possible to benchmark GPT4 in the near future, which is doubtful, maybe in 2025 or 2025... we may already be there, depending on what threshold you accept. https://github.com/manyoso/haltt4llm

@PatrickDelaney when I say Benchmark in the above comment, I mean run an inference of GPT4. Also see my concerns and questions below to Jacob Pfau about OpenAIs problematic habit of using benchmarking metrics in training. That being said GPT4 all and Llama are already significantly high.

@PatrickDelaney

Id recommend as benchmarks: human-eval code top-1, MMLU, and big bench hard.

@JacobPfau Correct me if I'm wrong, OpenAi ignores requests not to train on open datasets including big bench from what I have read so that would be invalid. Further, I'm not sure GPT4 is an inference model that OpenAI will submit to any leaderboards as it is proprietary? Lastly, we would have to compile Big Bench results ourselves based upon the current status of the repo, assuming that a test was even run?

If GPT-4 does some things (like specifically poetry) better, but it's widespread - understood that the new model is better at basically everything else, and by a margin, and nobody would consider using GPT-4 unless they wanted that niche ability - how would you resolve that?

@YonatanCale "I'll go with whatever popular consensus seems to indicate the top benchmarks (up to three) are regarding performance." --> if that condition is satisfied but there's one particular thing that's not well captured by the benchmarks (such as poetry, or performance in rap battles, or coming up with sufficiently delicious cheese soup recipes), that's fine an this still resolves YES.

So it doesn’t matter if the model can fit in VRAM, just that it runs inference on a consumer PC no matter how slow?

@EricG Yep. Just has to run inference on a consumer PC, and return a reasonable length message in less than a day. Run it on CPU if you have to, this market doesn't care.