In order to qualify to resolve this market to YES, I don't have to agree with the argument, it just needs to be a real attempt to argue in favor of that conclusion, without basic logical errors. A non-exhaustive list of such errors:
Assuming the probability you want without justification, such as "if we don't know a probability it must be 0" or "obviously the probability is negligible".
Magical or non-physicalist arguments like "computers are incapable of intelligence" or "God would prevent that from occurring".
Disproving one specific attack vector and treating that as having disproven all risk. "[20 pages explaining how an AI could not possibly use stuffed animals to exterminate humanity], and therefore AI isn't a risk!"
If I'm not aware of any by market close, this resolves NO.
I won't bet.
🏅 Top traders
| # | Name | Total profit |
|---|---|---|
| 1 | Ṁ104 | |
| 2 | Ṁ64 | |
| 3 | Ṁ52 | |
| 4 | Ṁ47 | |
| 5 | Ṁ43 |
I think there are perspectives I can't discount that have x-risk at <1% over 50 years because they think AI will develop slowly or because AI will, even if misaligned, try to work with humans in many specific cases in its narrow interests, or whatever. I definitely think it's unreasonable to be that confident in those prespectives given how terrible we are at predicting AI progress generally. I'm pretty confident you can make arguments that it's <1% that are wrong for reasons more complicated than the reasons most 'serious arguments' in any field are wrong. There's probably some post on lesswrong from someone who claims they think AI risk is significant but has long timelines that counts
@IsaacKing I thought this one counted by your market description.
https://optimists.ai/2023/11/28/ai-is-easy-to-control/
It's at least a "real attempt ... without basic logical errors".
@MartinRandall It wasn't really an argument against AI being risky, they were just pointing out that AI minds are more directly interpretable than human minds.
We will not attempt to directly address pessimistic arguments in this essay
If I were going to accept something like this, I'd also have to accept a two sentence argument like:
The vast majority of new technologies have made the world better for humans rather than worse, so our base rate on doom should be much less than 1%. This does not take into account arguments for why AI is different, I will address those later.
On the technical aspects of the source of risk, I doubt you could find any serious arguments; the presence of serious risk is basically the consensus of the top experts and the major AI labs (not to mention the world's governments and the general public). There are plenty of arguments from AI experts that human extinction wouldn't be a "risk", it's the desired result. There are also some arguments that humanity will never reach that far, either by global effort to shut down progress in the relevant technology, or by apocalypse some other way, and I wouldn't be surprised if some tried to argue that either of those are certainty. And, of course, there will be individuals with their own half-baked ideas for alignment that they're sure would work (and would be universally implemented), even if nobody else does.
the presence of serious risk is basically the consensus of the top experts and the major AI labs
lol no it's not.
@Snarflak ? The most prominent commercial AI labs are OpenAI, Anthropic, and DeepMind, and essentially the entire leadership of all three (including all their CEOs) signed the CAIS AI extinction risk statement. The three most-cited AI researchers in the world are Yoshua Bengio, Geoffrey Hinton, and Ilya Sutskever, who are all spending their time on preventing AI-caused extinction through advocacy, research, or both. Martin Hellman, Stuart Russel, Peter Norvig, David Silver, Dawn Song, Andrew Barto, and Ian Goodfellow also acknowledge the risk. Saying that there's no broad agreement because of Meta's Yann Lecun is like saying, "hey, we got some oil companies to pay this scientist to say that global warming's a hoax, so clearly it's a matter of dispute."
@Ebcc1 This worldview seems completely absurd to me. I want to bet "YES", but the way the bet is worded I suspect no argument would be considered "serious" enough for the host.
Only just now seeing this. I think it would be helpful for you to give examples of what would qualify in a different context—especially one where people have reasonable beliefs that the risk is higher than 1%. e.g., "We assess the risk that a nuclear explosion will ignite the atmosphere is <1% because we have a model here saying so." (Except the atmosphere ignition example has issues given that it was not credibly argued that it was >1% likely to happen)
@HarrisonDurland Yeah, that would count. Or for a current example, risk of a serious asteroid impact in the next 10 years.
Does human extinction that involves AI misuse of the "ask a chat model how to do something and follow the directions" count as "human extinction from AI"?
If yes, does "use AI for summarization of the information in a textbook" count?
Basically the question is what counts as "from AI". Because I think even if AI didn't exist it would be pretty hard to argue for <1% extinction risk over the next 50 years.
@FaulSname If GPT-5 can be asked for a computer program that, when run, exterminates humanity, and provides a correct answer to this question, and this happens, that would count as existential risk due to AI.
@IsaacKing My question is how much of the work the AI has to be doing to be counted as "from AI". I think that the chance of extinction risk due to human behavior, even without AI, would be > 1% over the next 50 years. Separately, I think that it's likely that within the next 10 years, most engineering tasks that people do will touch AI in some way -- e.g. I think one of the first (figurative) killer applications for AI will be writing and updating technical documentation of existing systems, which means that any engineering work that is likely to destroy the world will probably use AI for routine stuff like that.
I think I can make a pretty strong case for "Yudkowsky's FOOM is very unlikely, Christiano's Whimper is almost certain to take more than 50 years to result in human extinction." But if "a human wipes out humanity using a process that touched AI" counts then there's no way I can argue the risk is <1%.
@Snarflak depends on the particulars of how real e.g. nuclear winter is, speed of advancement in spaceflight, and how long it takes the last groups of humans to die out in a collapse of civilization scenario. But I don't see getting a number very far under 1% even without AI in the mix. I think the risk of just a full-scale nuclear exchange in the next 50 years is at least 10%, and extinction seems plausible as a worst-90th-percentile outcome there.
@FaulSname If it's primarily human-driven and the AI just helped a bit, that doesn't count. e.g. using AI to write documentation of nuclear weapon design, and maybe to design improved missiles which humans then decide to launch, doesn't count. The extinction has to be primarily done by the AI, in a way that humanity would not have done themselves.
@IsaacKing sure, but 1% total is <1% in 50 years, especially with the approach they are arguing for where there is never a provably safe time.
Alright, just read this. The authors definitely do a better job of avoiding blatant errors than most.
I don't think this really qualifies as an argument for the proposition that risk is <1% though. For example they frame jailbreaks as a positive, because the user is able to "control" the AI better than the company did. If this trend continues until superintelligence, some internet troll could simply jailbreak GPT-6 to provide the code to a paperclipper.
This an amazing line though.
People will look at you funny if you say that you’re studying methods of better controlling humans. As a result, human control researchers have to refer to themselves with euphemisms such as “marketing strategist” or “political consultants,”
@IsaacKing I think your "Internet troll" argument is a possible counter argument, but you can't reasonably expect a single paper to discuss every possible counter argument.