Current AIs like ChatGPT gain their capabilities by mimicking humans, and human capabilities are optimized to support human concerns. So I think the orthogonality thesis is nearly complete false for ChatGPT-style models.

However, I think mimicking humans isn't the only way to make AI, and that the orthogonality thesis will be much more true for other methods like consequentialism, whose goals are only limited by instrumental convergence.

@tailcalled This seems to be an argument against the strong orthogonality thesis, which roughly holds that: for any given environment that produces an agent, the agent's intelligence and values are orthogonal. But it sounds like this market is talking about the weak orthogonality thesis, which is more accepted. The weak orthogonality thesis roughly says that: in principle, almost any level of intelligence could be compatible with almost any goal.

The paper linked to in the market description also provides a definition of the orthogonality thesis along these lines:

Intelligence and final goals are orthogonal axes along which possible agents can freely

vary. In other words, more or less any level of intelligence could in principle be

combined with more or less any final goal.

This is entirely correct, for example liberals support government then oppose the functionaries through which it has any power whatsoever (cops). Not defending cops just showing that linear, first order reasoning creates contradiction regardless of the strength of the reasoning and these contradictions eventually destroy any system they attempt to build.

@MarkIngraham That's... not what the orthogonality thesis is.

@IsaacKing it shows the intelligence of ai and its goals are unrelated, ie, it can attack its own functionaries while pursuing first order goals. It's like a ant colony eating itself.