Many people are worried that in the next few decades humanity will create a superintelligence which brings extinction, enslavement, or some other terrible fate upon us in "AI Doom". This question asks how we avoided this by 2060, in the worlds that we did.
Please try to keep answers short and specific. Describe one thing, and describe it in a way that the average manifold user can understand. Don't assume your audience knows a bunch of very technical terms. Try not to present answers in a biased way like saying "We are lucky that [thing you think is unlikely] happens." Just say [thing happens].
If you have multiple things to say, say them in multiple submissions. Make as few assumptions as possible in each individual submission. You can elaborate in detail in the comments. It's better to submit something that doesn't overlap with existing answers too much, but submitting a much better version of an existing submission is also okay.
This question is one in a long line of similar questions in various formats. I think we mostly expect that humanity will not survive to resolve these questions, so they mainly represent the opinion of people willing to lock their mana up indefinitely. They also represent the opinion of people trading on other people's opinions in the short-term.
This questions tries a new way to incentivize long-term accuracy. In about a week, maybe a bit sooner if this question doesn't get much interest or a bit later if it gets more interest, this question will close. Then all answers will resolve N/A. All trades, profits, and losses will be reversed, and all mana returned.
If we all die and/or manifold ceases to exist, your reward will be the trader bonuses you got for submitting interesting answers. You'll also have bragging rights if your answer was voted high up before market close. I may also award bounties to very insightful answers.
If we survive until 2060, then I or another moderator will use the "unresolve" feature to undo the N/A resolution and put everyone's mana back into the market as it was at market close. All answers will then be graded for their general quality by the best experts I can find, and will resolve to % from 0 to 100. There will be a grading curve, so one can expect the answers to be graded relative to each other instead of being compared to a hypothetically perfect answer which was not submitted.
Hopefully, this format will function like a poll that is weighted by the amount of mana that people have and are willing to spend on it, and will produce more accurate results than an unweighted poll.
Please do not submit answers that are too similar to existing answers, or which are just bad jokes. I will N/A answers that I think are not worth the space they take up in the market. Good jokes may be allowed to stay up longer than bad jokes. This market will be unranked, so as not to disrupt Manifold Leagues.
I am open to suggestions for improving this format, and may update these rules within the spirit of the question.
Related questions
Thanks for participating everyone! And sorry about the emails, wish those were grouped all together for unlinked multichoice markets.
This was a weird experiment, I'm not sure it generated that much insight but hopefully some value was gained nonetheless. Fingers crossed, I'll see you all in 2060 for the proper re-resolution! In the meantime, my new weird experiment of the week can be found here:
This market structure basically gives no consequences for shit takes, especially beliefs that people can profit off of IRL, like LeCun's "AI is a tool and always will be", or whatever shameless garbage the eleuther people are peddling this time around.
Yud's original market won't resolve but will still move, rewarding and punishing people based on how reality swings as the AI situation unfolds, and as more people join manifold and develop quantified track records on specific topics, and predictive bots move markets based on predictive analytics based on those quantified track records.
It's a fantastic idea but needs iteration, if there is a viable final finished form this is not it. Looking forward to participating in the next attempt too.
Don't loans mostly solve the problem of long-term trades tying up funds?
If I'm doing my calculations correctly, if you simply left this open till 2060, then in (for instance) 2 months time, assuming we all take our daily loans, then this market will only hold ~9% of our original investment, with the rest of it loaned back to us (and more to return to us over time).
Granted, you've designed this to return our mana to us faster than that, but I'm not entirely convinced that was necesarry.
@Shump An unaligned AI may be able to understand what humans want "well enough". When we say alignment, we generally do not mean perfect accordance with human values. After all, humans do not know what we want and often disagree with each other. We mean something more like "don't kill everyone", and the issue is so much not understanding what that means as it is having strong assurances that the AI will act in accordance with its understanding of that in all possible scenarios. So the unaligned AI may get the concept of "don't kill everyone" and come up with clever ways to architect AI solutions that respect it along with convincing ways to prove it.
You could give the task to a superintelligence, but in that case you already took a big risk and the superintelligence might have killed you.
@aashiq I'm sorry but I remain convinced that AI risk people do not understand how Machine Learning works. That's not how any of that works... But anyways, that's a long discussion and this is not the place for it.
@Shump FWIW, I agree with you in general disdain for them. Please consider the above to be my attempt to model their fears, rather than my own beliefs
@Shump There's no reason an AI needs to be aligned to human values to solve arbitrary problems which includes aligning an AI to any set of values.
@Shump - I don't think AI risk people are necesarrily all relying on current Machine Learning techniques being what gives rise to AGI.
@MatthewLeong Valid, but that means that AGI would require a paradigm shift in AI. Current Machine Learning makes machines that are trained to solve specific tasks in a way that probably won't fit whatever AGI means.
@Shump I don't think current text models are trained to solve specific tasks. Communicating by text is a very general task.
@MartinRandall They are specifically trained to generate the next word in a way that fits human-written language (pretraining) or whatever humans like to see (RLHF). That's all they do.
They are specifically trained to generate the next word
The next token, if we're being specific. A token can be all sorts of things, which is why LLMs can do things like write programs, and play chess.
@aashiq > After all, humans do not know what we want and often disagree with each other.
This just means humans are unaligned, not that the problem is easier because it is easier/simpler for you to imagine it that way...
@Joshua https://manifold.markets/Alicorn/eliezer-yudkowsky-stock-permanent
The Yud permanent stock should probably have some relationship with this choice, and have more traders in general
@DavidJohnston A good point! I feel like the term is used enough that most people on manifold should be thinking of the same order of magnitude of intelligence, but I'm sure there are still many disagreements.
I'm open to putting an exact definition in the description if people have suggestions.
@Joshua The kind of case I worry about (resolution wise) looks like sustained widespread use of automated science and technology, but it’s basically recognisable as an extremely competent version of what people do today
@DavidJohnston I wouldn't call that superintelligence. And even if OpenAI comes out with GPT 6 and calls it superintelligence as a marketing gimmick and the term gets diluted, this market will still resolve according to the idea of superintelligence that most people use when talking about AI Risk.
If it's not intelligent enough that we have reason to worry that it could kill us all, I wouldn't call it a superintelligence. Still open to a more formal definition though.
@Joshua Let's say GPT-6 come out, and it's a multimodal LLM that's better than humans at every single task within its text, audio, and visual capabilities. Is that superintelligence? This hypothetical GPT-6 still won't be able to do things like "resist shutdown", "feel love", or "make friends" because these are things that LLMs are simply not designed to do.
I guess by your definition that's not superintelligence becuse it's not capable of killing us all. But that's kind of a weird definition. An augmented human might be superintelligent, but not capable of killing everyone.
@Shump I'm not saying I'm defining superintelligence as something that actually is capable of killing us all, just that it's smart enough that we have reason to worry.
Wikipedia just defines it as "a hypothetical agent that possesses intelligence far surpassing that of the brightest and most gifted human minds", which I think is enough of a definition for the market?
We can leave it to the far future resolvers to fairly decide what counts as "far surpassing", I think.
@Joshua - "If it's not intelligent enough that we have reason to worry that it could kill us all, I wouldn't call it a superintelligence."
I do like that worknig defintion, at least for this question.
Like, avoiding AI doom because the candidate superintelligences are not artificially superintelligent enough to cause doom, just sounds like not even facing the risk of AI doom yet and having not made a superintelligence.