Related questions
Experts are greatly debating on whether the AI can wipe out humanity by 2100. The information given above indicates that there are opposing views. According to the Existential Risk Persuasion Tournament, AI proponents calculated a probability of about 3% that artificial intelligence will exterminate humanity within the next hundred years; however, “super-forecasters” gave the odds as low as 0.38%.
These are just estimates, however, based on existing knowledge and perceptions. The future evolution and consequences of AI rely upon several factors involving technology updates, ethics, legal systems, and how people welcome its use.
Researchers, medical practitioners and policy makers are working closely with their organization's to make sure that AI is developed responsibly and safely. The potential risks surrounding robust AI-based systems exist, but predicting their impact on mankind in the long run remains an impossibility.
@jack I think the probability that AI kills all humans conditional on killing at least 10% of them is upwards of 90%, so my incentive on this market is still to bet it down pretty low even though I think the actual probability is much higher.
This is better for people who assign a lower conditional probability though.
Metaculus has human extinction from all sources at only 1.5%. https://www.metaculus.com/questions/578/human-extinction-by-2100/
@MaxMorehead Metaculus predictions on this particular question, and on several other related questions in the topic, are terrible for a variety of reasons, it's been discussed before.
For a more reasonable forecast, check out https://samotsvety.org/blog/2022/09/09/samotsvety-s-ai-risk-forecasts/ which forecasts 25% chance of misaligned AI takeover by 2100.
@jack That aggregation forecasts "AI takeover". I'm not sure exactly what their definition of that term is (do I have to read WWOTF to get it?), but I assume it incorporates situations in which humans lose control of the future but are nonetheless not fully extinct.
Secondly, I included the Metaculus prediction not necessarily to argue that it's right, but argue that the current odds in this market are ridiculously far off from "mainstream" opinions. I'd think that the average Metaculus user would be more concerned with existential risk than the average knowledgeable person, and Metaculus generally predicts fairly fast AGI timelines, and still the prediction on AI-caused extinction is not even close to 15%.
The problem with forecasts on AI-risk from within the EA AI safety community is that there is a strong selection bias. The only people who "have done the equivalent work of reading the Carlsmith report" are people who are already quite concerned about AI ex-risk. The strain of thought that says "AI is likely to kill us all" has not yet had a large enough effect on company and government policy to merit well-researched responses from people who disagree with it (though that's probably changing).
In my opinion, without getting into particular arguments, I'd expect the "mainstream" to vastly underestimate AI ex-risk, but I'd also expect people in EA-AI circles to systemically overestimate the risk. So, I think the fact that even the prediction you linked makes 15% look like a fair price (when incorporating the probability of non-extinction takeover, and pre-TAI catastrophe) is motivation for me to buy NO.
@MaxMorehead To set context, my goal here is to predict the questions we care about, it isn't to try to bet this particular market to the "correct price" (there are many reasons the price here is not a reflection of predicted probability, in any case). Also, if the probability is even 1%, we should be taking it massively more seriously than we are now.
I don't know the exact definition of AI takeover but yes, it certainly includes loss of control without extinction. I think that is the more relevant question, as most such scenarios are also very bad for humanity, I believe.
Yes, the predictions of the mainstream on this question are very low. Is it worth considering? Yes. Are they right? I think not. A lot of the quality of my predictions comes from deferring to other forecasts, and judging the correct amount to defer to them. I put very little stock in the "mainstream" take here. The typical mainstream portrayal of the issue is such a poor caricature of the real issues, imo.
The predictions from EAs who don't work on AI safety are also high, it's not just the AI safety folks.
No, the notion that AI will lead humanity to extinction in such a short period of time is highly unlikely and largely a topic of science fiction.
Researchers, policy makers, and organizations are actively working on ethical AI development, safeguards, and regulations to ensure that AI technology isused responsibly for the benefit of humanity. While AI hasits challenges, it also offers opportunities to solve complex problems and improve our quality of life if used wisely and with appropriate safeguards.
@26a0 I too would predict no, but none of those regulations and ethical guidelines mean anything when anyone can have access to an open-source model of just about any kind (which is increasingly becoming true).
@26a0 We agree with the analysis and would have made the same bet. I believe it is essential to approach this question with a nuanced perspective as done in the comment.
According to an article on Freethink.com, “an Existential Risk Persuasion Tournament, held by the Forecasting Research Institute collected predictions from technical experts and “super forecasters” (people with a track record of making accurate predictions) about various existential threats — ranging from AI to nuclear war to pandemics.) And the threat of AI was the most divisive topic. AI experts said there is a 3% chance that AI will kill humanity by 2100, according to their median response. Superforecasters were far more optimistic, putting the odds of AI-caused extinction at just 0.38%. The groups also submitted predictions on the likelihood that AI will cause a major catastrophe, even if it doesn’t wipe us out entirely (defined as causing the death of at least 10% of humans over a 5-year period). “
As seen from the above article, there has still been no clear consensus among experts regarding the timeline and likelihood of AI causing harm to humanity. Some experts believe that AGI (Artificial General Intelligence) with the potential to pose existential risks may not be realized for many decades or even centuries.
Additionally, AI has played an instrumental role in bringing about healthcare advancements. AI’s development is not solely focused on destructive capabilities.
“In summary, predicting whether AI will pose an existential threat to humanity by 2100 is uncertain and highly debated. While there are valid concerns, there is also a strong emphasis on responsible AI development and ethics. The future of AI's impact on humanity will depend on the decisions and actions taken by researchers, developers, policymakers, and society as a whole.” (Forecasting Existential Threats, 2023)
By Team 4: Andrés Tocaruncho, Andrea Montserrat Alvarez Rodriguez, Cesar Ruiz, Chengran Shi, Guilherme De Souza, Komal Ghazanfar
@GG Unresolvable markets tend to become Keynesian Beauty Contests towards the default option, I guess?
@jeremiahsamroo You bet that there is a 42+% chance that humans get wiped out this century... Bad bets indeed cause you to lose mana.
@FlorisvanDoorn I was market making when the influx of LK-99 members caused this market to dip. I held onto my investment bc of the sunk cost fallacy. I am very anti AI doom.
@jeremiahsamroo 2100 is a long time for it to surge again, you can just keep HODLing :P
@jeremiahsamroo Most people are anti AI doom. It really needs a rebrand to a friendlier name.
@IsaacKing says the probability on this market is "obviously wrong".
/IsaacKing/will-manifold-stop-having-obviously
Will this move the market?
@MartinRandall Don’t think it’s so wrong anymore. It used to be very wrong at 45%.
@ShadowyZephyr Yeah, I agree. I'd put the probability at 10%, but predictions this far out have a huge margin of error and I don't think 17% is unreasonable. 45%, though...
Even 45% doesn't seem "obviously wrong" imo. Many forecasters think that it is far, far easier to develop an unaligned superintelligence than an aligned one, and that most unaligned superintelligences would have catastrophic consequences for humanity, and that a superintelligence is likely to be developed by 2100.
I'm a bit more optimistic than 45%, but not that much more.
I meant "obviously wrong" in the sense of economic incentives, not actual predictions. The correct probability for this question in a market of homo economicus is 0%, regardless of how likely AI actually is to wipe our humanity.
@jack So is it fair to assume you think AI development should be paused indefinitely?
@IsaacKing Sure, in a world of perfectly rational economic actors (plus some other assumptions about things like not caring about information signaling etc), this market wouldn't exist because nobody would bother to buy YES. But I don't think that's the point here, and that's different from saying that the economic incentives are all to NO - because the real world is not that world. It is true that one nash equilibrium for this question is 0%, but it is not the only equilibrium, and it is not the one we should expect to see in the real world.
@BTE I think we should be investing massively more in AI safety. I think pausing AI development is impractical; I'm unsure whether or not it would be a good idea in theory but in practice it doesn't really matter.
@jack If Manifold were to suddenly catch on among wall street traders, this question would rapidly converge to 0. The fact that it hasn't is an idiosyncrasy of Manifold's current user base, and won't reflect anything generally true about the world other than by coincidence.
@IsaacKing It would likely do that but it's by no means certain - look at meme stocks as a counterexample. The "idiosyncrasy" isn't limited to Manifold, it's more general than that.
@IsaacKing We do have a few quants and other sophisticated traders and they are not buying this market down to 0%. Mira and others have explained some of the reasons why below.
@IsaacKing In addition to what Jack said, there's some chance that AI wipes out humanity and there continues to be a chain of transactions between agents who care about Manifold resolutions, so I don't agree that the "correct" solution is quite 0% even given your particular choice about which perverse incentives to model. That said, I agree that the "correct" probability in your sense is probably a lot lower than the "true" probability of the event happening. I have other disagreements with your notion of the "correct" probability that I think are more major; I'm still thinking about how to present them in a way that has net good consequences.
@MartinRandall While I agree there's various reasons this market won't hit 0 prior to resolving, I think liquidity probably has something to do with that as well. If I wanted to buy this market down to the percentage I thought was accurate - around 10% - I'd need to spend approximately one million mana, or $10,000. I would need to spend two thirds of my yearly graduate TA stipend on that. I'd need to be a multimillionaire before I ever even thought about spending that much on Manifold. Betting this market to zero would take $29,000. Similarly, I think that the infamous AI-kills-everyone-by-2030 market should be at 1%, but I can't afford to spend hundreds or thousands of dollars to bet it that low.
Meanwhile, if I wanted to completely fuck up the market's probability and set it to 50%, it would "only" take 100,000 mana. It would be ten times cheaper for me to set this market to a percentage that I think is absurd than it would be for me to set it to a percentage that I think is accurate. I also wouldn't be making great returns on NO since the price is already so low.
tl;dr The structure of Manifold makes it such that small, highly energized groups (e.g. AI doomers) can stop the probability of markets about unlikely events from reaching the true probability, because buying YES at low percentages (or NO at high percentages) is cheap, and the payoff can be huge (though in this case, what payoff...?)
Related questions
