Do transformer language models prefer superposition even when number of neuron dimensions available > input features?

10

210Ṁ940

resolved Jan 1

Resolved

YES

1H

6H

1D

1W

1M

ALL

Mechanistic interpretability

New Year's Resolutions 2024

Get

1,000

to start trading!

🏅 Top traders

#	Name	Total profit
1		Ṁ73
2		Ṁ1
3		Ṁ0
4		Ṁ0

Sort by:

I don't understand what this question means. 98% seems ridiculously high, but maybe you mean eg "features are not neuron aligned"? And I don't know the resolution criteria (I also avoid betting in markets where the creator has a massive yes position and unclear criteria!)

People are also trading

If LMs store info as features in superposition, does # features scale superlinearly with number of model parameters?

Will Transformer based architectures still be SOTA for language modelling by 2026?

Are Mixture of Expert (MoE) transformer models generally more human interpretable than dense transformers?

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

Will superposition in transformers be mostly solved by 2026?

Do you think Mixture of Expert (MoE) transformer models are generally more human interpretable than dense transformers?

If LMs store info as features in superposition, are there >300K features in GPT-2 small L7? (see desc)

Will we find polysemanticity via superposition in neurons in the brain before 2040?

Related questions

If LMs store info as features in superposition, does # features scale superlinearly with number of model parameters?

Will Transformer based architectures still be SOTA for language modelling by 2026?

Are Mixture of Expert (MoE) transformer models generally more human interpretable than dense transformers?

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

Will superposition in transformers be mostly solved by 2026?

Do you think Mixture of Expert (MoE) transformer models are generally more human interpretable than dense transformers?

If LMs store info as features in superposition, are there >300K features in GPT-2 small L7? (see desc)

Will we find polysemanticity via superposition in neurons in the brain before 2040?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules