
Following on this useful input from Steven, I shall clarify my resolution criteria.
Steven: Does this resolve NO if AI recursive self-improvement results from a gradual increase in generality?
Nathan: I say 'allows for' recursive self-improvement (FOOM-after-deployment). It might be superhuman right out of the box (FOOM-during-training). I will resolve YES if it becomes strongly superhuman by any means at any speed, and humanity manages to solve AI safety via alignment or governance sufficiently to prevent catastrophic conflict. I will also resolve YES if there is a conflict and the AI is clearly winning or has won (if I'm still alive to do so).
Related:
Thomas Kwa: How does this resolve if the most likely AI failure mode is disempowerment that doesn't cause human extinction (as in What Failure Looks Like part 1)?
How does this resolve if takeoff is slow or medium speed but AI is still an existential risk?
Nathan: I deliberately stated 'helpless', as in disempowered, rather than extinct because I wanted to include the broader case of either outcome.
AGI achieving clear broadly superhuman capability before 2031 and disempowering humanity counts as a YES for this market, regardless of the shape of the curve describing the increase in capability.
Steven: What if it results from crossing a competence threshold while remaining at a similar level of generality?
Nathan: answered above
Steven: What if AI quickly attains superhuman capability but not because of recursive self-improvement?
Nathan: answered above
Steven: Are there any conceivable cases where you'd judge the first sentence to be true but the second sentence to be false? ("Solve safety or be unsafe" would almost be a tautology, but I guess "helpless" is stronger than "unsafe", and this is meant to resolve NO on cases where AI is dangerous but humanity still has a fighting chance, and cases where humanity has to solve safety but can afford to do so after superhuman AI capability?)
Nathan: If AI is actively working on taking over but I judge that humanity still has a fighting chance, I will resolve NO. I believe this would falsify the second part of the claim, "Before that point humanity must solve AI safety or be helpless." Humanity, in this case, has neither solved AI safety nor become helpless. In this case, the world would be clearly at war, lots of people would be dying, lots of economic damage would be occuring. I'd call this a 'Skynet scenario'.
If superhuman AI is created but humanity has managed to prevent its loss of power through the end of 2030 due to successfully constraining the AI (e.g. via governmental action to enforce safety laws prevent the release of the AI), I will resolve YES. I will count this as 'temporarily solving safety' even though I don't count it as 'solving alignment' or establishing long-term safety (since other groups could develop and release bad AI, or the containment could fail). It's an unsatisfying YES, I admit. If there is a war, but humanity wins and the war is not currently ongoing as of 2030, it will also count as a YES for this reason.
I appologize for the complication resulting from making multiple claims in one market, but I did indeed want to get at the multi-claim question here.