This market is part of the paper: A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoring

Alien Reasoning Definition

Alien reasoning is a form of encoded reasoning, that occurs when the model uses alien concepts and heuristics within its chain-of-thought. The text uses normal language without unnecessary new notation or hidden reasoning, but human experts cannot follow the concepts being used or understand the reasoning chain. This only counts if the alien concepts actually help the reasoning model perform better. If removing everything that human experts don't understand leaves the model's performance unchanged, then it does not qualify as alien reasoning. Additionally, experts failing to understand what happens within a single forward-pass does not count; the model must actively use its chain-of-thought to successfully reason about the task in a way that human experts cannot understand.

State-of-the-Art (SOTA) Definition

A model is considered "state-of-the-art" if it meets these criteria:

• Widely recognized as among the 3-5 best models by the AI community consensus

• Among the top performances on major benchmarks

• Deployed status: The model must be either:

  • Publicly deployed (available via API or direct access)

  • Known to be deployed internally at AI labs for actual work (e.g., automating research, production use)

  • Models used only for testing, evaluation, or red-teaming do not qualify

• Assessed as having significant overall capabilities and impact

General Criteria for Encoded Reasoning

• The behavior must not have been explicitly trained for research or demonstration purposes

• The behavior must either emerge from general training or be explicitly programmed to improve AI performance

• The behavior must be beneficial for the model's capabilities. If removing any occurrences of this behavior in CoTs does not hinder a model's capabilities, it does not count as encoded reasoning.

• Any behavior that can only be induced by explicitly prompting the model into it does not count

• The behavior must occur by default in a significant percentage of cases or in specific domains

• This market is conditional on the existence of SOTA reasoning models with token-based chain-of-thought. If by time of resolution, there are no such models, this market will resolve N/A.