Could you provide a link with context?

@jonsimon https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=C4SK_El5WKWCmowYGCvYMUBy

Or

Source -

https://www.lesswrong.com/posts/euam65XjigaCJQkcN/an-analogy-for-understanding-transformers

An Analogy for Understanding Transformers - LessWrong

Thanks to the following people for feedback: Tilman Rauker, Curt Tigges, Rudolf Laine, Logan Smith, Arthur Conmy, Joseph Bloom, Rusheb Shah, James Dao. …

@firstuserhere Thank you for the LW link, I'll give it a read. I'm familiar with what residual connections in ANNs are, and the high level "Idea" at least doesn't make a lot of sense to me since it's not true that information is only transmitted via Residual connections. (Just because the information is processed in some way when transmitted over other connections doesn't mean it's not there.) So I'll bet No for now and possibly switch after reading the full post.

@jonsimon Better reference: https://transformer-circuits.pub/2021/framework/index.html#subspaces-and-residual-stream-bandwidth Note that this is a conceptually shift from the standard way of thinking about residual connections as "skip connections", and instead thinks fo the residual stream as a shared object that each layer reads from and incrementally writes to.