Skip to main content
MANIFOLD
Will superposition just go away if i keep increasing the number of MLPs in a 2L transformer language model?
2
Ṁ70Ṁ15k
resolved Jul 11
Resolved
NO

up to d_mlp = 100X d_model

Market context
Mechanistic interpretability
Get
Ṁ1,000
to start trading!

🏅 Top traders

#TraderTotal profit
1Ṁ47
2Ṁ19
Sort by:

like neuron superposition instead of residual stream superposition, right?

People are also trading

Will superposition in transformers be mostly solved by 2026?
46% chance
If LMs store info as features in superposition, does # features scale superlinearly with number of model parameters?
41% chance
End of pre-training era for language models: Will an LM fine-tune for more FLOPs than it is pre-trained for, before 2026
37% chance
[Carlini questions] Will we still use (slight modifications of) transformer-based LLMs we currently use
When will a non-Transformer model become the top open source LLM?
Will Transformer-Based LLMs Make Up ≥75% of Parameters in the Top General AI by 2030?
50% chance
Will Scaling Laws for Neural Language Model continue to hold till the end of 2026?
82% chance
If LMs store info as features in superposition, are there >300K features in GPT-2 small L7? (see desc)
59% chance
Will second-order optimizers displace first-order optimizers for training LLMs by 2030?
43% chance
Will Scaling Laws for Neural Language Model continue to hold till the end of 2027?
68% chance