I suspect Chinese companies will soon do better at answering questions in Chinese, and US companies will continue to do better at answering questions in English.

I know this isn't what the question means but currently since they have downloaded llama 3.1 they are already competitive with SOTA.

China will (and already has) released good models, but I doubt they will optimize for the US market as much as US companies do. OAI and others are already optimizing for the benchmark, while Chinese companies need to keep in mind their own domestic market and it's rules.

Always put an extreme limit order no matter how ridiculous it seems...

I will not make this mistake ever again

https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct

quant ppl are good at data cleaning

Leading on lm leaderboard seems like a tall order. It's really dependent on release schedule, and both Google and OpenAI have bigger models either under review or in training

I am somewhat surprised that the market didn't update more on Claude 3.5 Sonnet (since Opus was quite low and the resolution criteria were quite sensitive to this).

Curious as to why people are going bullish in this market.

It's likely that if the US starts to feel threatened by China in the AI race they will start restricting the flow of compute power into China which is a crucial component of AI. The US is ahead by a large factor in terms of graphics cards R&D - think NVIDIA and AMD.

@idk Well, first, Chinese companies were successful building frontier models with only H800s. Also, the Ascend GPU from Huawei are starting to show some success.

The fact that Qwen trained a good 72B model tells us that as soon as they change from dense to MoE which uses the same about of computing power, they should get 4t level model. This already happened to Yi-Large.

@idk They are already doing that, and it wouldn't change things that fast.

https://qwenlm.github.io/blog/qwen2/

@SimranRahman Qwen2 72b probably will fall short of Claude Opus, but imagine Alibaba uses the same technology and scale to a proprietary MoE with 72B active paramter and 400B+ total parameters. Easily > Opus

@Sss19971997 Correct, thinking along these lines also

in the "hard prompt" category a chinese model is already competitive (not the general arena which I assume the resolution hinges on)
https://lmsys.org/blog/2024-05-17-category-hard/

(the votecount is low, maybe it will change but good news for the YES holders)

@Ledger If a Chinese model surpasses Google's model, but not Anthropic's or OpenAI's, does this resolve YES?

@jacksonpolack https://manifold.markets/Ledger/will-china-be-competitive-in-the-ll#8vbkwsqiy8g

Yi-XLarge is in the oven cooking right now.

@Sss19971997 Yi-Large already score 51% on Alpaca-Eval2

@Sss19971997 Yi-Large ranked 1236 on Lmsys

due to sheer numbers of engineers there are almost certainly CCP agents inside google and as of 11 years ago when I worked there almost all the source code is visible to almost all employees so it would be pretty easy for them to steal the code eh?

@JonathanRay tho the code is probably unusable outside google because they use their own special snowflake infrastructure for everything. A lot of it would have to be rewritten

@JonathanRay Please look into how much compute, experimental runs and cleaned data is needed, and how hard building a large language model at the scale of Opus, GPT4, Gemini is (it's a lot, and it's very hard). Nevertheless I believe that by EOTY 2024 best Chinese ML researchers can pull that off, their Qwen 1.5 line is quite good. It's not as simple as get training code -> train.

@MADGAMBLER6969 Stealing the code and having the government pay for a zillion GPUs would go a long way

@JonathanRay efficient training code is architecture specific (what interconnects you have, what makes the calculations), most models right now are on the neox/llama base and those codebases are open source, saying that China steals tech is generally true but there isn't anything to steal in LLMs, it seems like there are very few secrets in how to make a good model, it's just very difficult and needs scale (but scale alone is not enough). Zillion GPUs is also a fantasy because there are more important places for compute (that today costs premium because of the bans on imports of advanced silicon) to be allocated in China, still some labs (Alibaba and Sensetime are main ones) have compute to participate in that dickwaving contest and I think they will overcome Google (they have a pretty bad track record for being leaders in cutting edge tech).