Before 2028, will there be a major self-improving AI policy*?

Question

Background.

*Resolution conditions (all must apply):

An AI of the form P(action|context) rather than e.g. E[value|context, action] must be a part of a major AI system. For instance language models such as ChatGPT or Sydney would currently count for this.

The aftermath of its chosen actions must at least sometimes be recorded, and the recordings must be used to estimate what could have usefully been done differently. RLHF finetuning as it is done today does not count because it solely involves looking at the actions, but e.g. I bet the Sydney team at Bing probably had internal discussions about this incident and those internal discussions would count.

This must be continually used to update P(action|context) to improve itself.

Criteria 2 and 3 must be handled by the AI itself, not by humans or by some other AI system. (This means that Sydney wouldn't count, nor would any standard actor-critic system.)

It does not have to be reflective, i.e. it do not have to consider the aftermath of its self-improvements and improve its self-improvement. Improvements to its self-improvement are allowed to be handled by people manually, by RLHF, by actor-critic methods, or lots of other options.

I will not be trading in this market.

Manifold Markets · Answer

Likely — Manifold Markets prediction market estimates a 74% chance (13 traders, as of Aug 6, 2025).