If Artificial General Intelligence has an okay outcome, what will be the reason?
454
5k
20k
2200
25%
Something wonderful happens that isn't well-described by any option listed. (The semantics of this option may change if other options are added.)
14%
I. The tech path to AGI superintelligence is naturally slow enough and gradual enough, that world-destroyingly-critical alignment problems never appear faster than previous discoveries generalize to allow safe further experimentation.
10%
K. Somebody discovers a new AI paradigm that's powerful enough and matures fast enough to beat deep learning to the punch, and the new paradigm is much much more alignable than giant inscrutable matrices of floating-point numbers.
9%
B. Humanity puts forth a tremendous effort, and delays AI for long enough, and puts enough desperate work into alignment, that alignment gets solved first.
9%
M. "We'll make the AI do our AI alignment homework" just works as a plan. (Eg the helping AI doesn't need to be smart enough to be deadly; the alignment proposals that most impress human judges are honest and truthful and successful.)
8%
L. Earth's present civilization crashes before powerful AGI, and the next civilization that rises is wiser and better at ops. (Exception to 'okay' as defined originally, will be said to count as 'okay' even if many current humans die.)
7%
O. Early applications of AI/AGI drastically increase human civilization's sanity and coordination ability; enabling humanity to solve alignment, or slow down further descent into AGI, etc. (Not in principle mutex with all other answers.)
5%
C. Solving prosaic alignment on the first critical try is not as difficult, nor as dangerous, nor taking as much extra time, as Yudkowsky predicts; whatever effort is put forth by the leading coalition works inside of their lead time.
5%
A. Humanity successfully coordinates worldwide to prevent the creation of powerful AGIs for long enough to develop human intelligence augmentation, uploading, or some other pathway into transcending humanity's window of fragility.
3%
J. Something 'just works' on the order of eg: train a predictive/imitative/generative AI on a human-generated dataset, and RLHF her to be unfailingly nice, generous to weaker entities, and determined to make the cosmos a lovely place.
1.5%
H. Many competing AGIs form an equilibrium whereby no faction is allowed to get too powerful, and humanity is part of this equilibrium and survives and gets a big chunk of cosmic pie.
1.4%
N. A crash project at augmenting human intelligence via neurotech, training mentats via neurofeedback, etc, produces people who can solve alignment before it's too late, despite Earth civ not slowing AI down much.
1.2%
G. It's impossible/improbable for something sufficiently smarter and more capable than modern humanity to be created, that it can just do whatever without needing humans to cooperate; nor does it successfully cheat/trick us.

An outcome is "okay" if it gets at least 20% of the maximum attainable cosmopolitan value that could've been attained by a positive Singularity (a la full Coherent Extrapolated Volition done correctly), and existing humans don't suffer death or any other awful fates.

This market is a duplicate of https://manifold.markets/IsaacKing/if-we-survive-general-artificial-in with different options. https://manifold.markets/EliezerYudkowsky/if-artificial-general-intelligence-539844cd3ba1?r=RWxpZXplcll1ZGtvd3NreQ is this same question but with user-submitted answers.

(Please note: It's a known cognitive bias that you can make people assign more probability to one bucket over another, by unpacking one bucket into lots of subcategories, but not the other bucket, and asking people to assign probabilities to everything listed. This is the disjunctive dual of the Multiple Stage Fallacy, whereby you can unpack any outcome into a big list of supposedly necessary conjuncts that you ask people to assign probabilities to, and make the final outcome seem very improbable.

So: That famed fiction writer Eliezer Yudkowsky can rationalize at least 15 different stories (options 'A' through 'O') about how things could maybe possibly turn out okay; and that the option texts don't have enough room to list out all the reasons each story is unlikely; and that you get 15 different chances to be mistaken about how plausible each story sounds; does not mean that Reality will be terribly impressed with how disjunctive the okay outcome bucket has been made to sound. Reality need not actually allocate more total probability into all the okayness disjuncts listed, from out of all the disjunctive bad ends and intervening difficulties not detailed here.)

Get Ṁ600 play money
Sort by:

Enough humans survive to rationalize whatever the outcome is as good, actually, and regardless of cost 90%+ of society treats anyone who points to visible alignment failure as a crazy person

If you write an argument that breaks down the 'okay outcomes' into lots of distinct categories, without breaking down internal conjuncts and so on, Reality is very impressed with how disjunctive this sounds and allocates more probability.

At the time of writing this comment, the interface tells me that if I spend M3 on this answer, it will move it from 0% to 10% (4th place).

I just wanted to check that this is right given that the mana pool is currently around M150k, subsidy pool ~M20k.

(The other version of this question seemed to have a similar issue.)

Why is this market structured with mutually-exclusive options? These don't seem remotely mutually exclusive. Indeed I would be pretty surprised if, conditional on survival, we didn't get some mixture of several of these options.

@MugaSofer Per the inspiration market: 'It resolves to the option that seems closest to the explanation of why we didn't all die. If multiple reasons seem like they all significantly contributed, I may resolve to a mix among them.'

Chunks of several buckets that add up to 'we got lucky and live in a universe where cooperation is mostly convergent under most starting conditions.'

I just upgraded this old market type to the new fixed payout version! Let me know if you see any bugs.

Humans are not the only weaker beings on earth and if ASI decides to protect weaker beings, at least we all get to be vegans in a vastly nicer economy.

Trying a new format for this question where you get all your mana back in a week, but then if we survive your bets actually pay off in 2060. Will it work? Let's find out!

Lots of people talking and arguing about AGI, but nobody's talking about how this market resolves.

One might naively think that we need AGI and it's unlikely to resolve at all, so this market is more of a discussion stand than a real market. But note the description states:

you get 15 different chances to be mistaken about how plausible each story sounds; does not mean that Reality will be terribly impressed with how disjunctive the okay outcome bucket has been made to sound. Reality need not actually allocate more total probability into all the okayness disjuncts listed

This doesn't read like someone actually planning to resolve to A-O + Other, and there happens to be 2 meta-options. And probability should correspond to "this market can possibly resolve to those options." So let's look at the 3 options other than A-O and see if they can resolve:

Something wonderful happens that isn't well-described by any option listed. (The semantics of this option may change if other options are added.)

This is the "Other" option, getting 11%. The market can't resolve to this option, or A-O, because it requires getting "at least 20% of the maximum attainable cosmopolitan value that could've been attained by a positive Singularity" and Manifold will shut down by the time that happens. So we can scratch this one off, leaving 5% and 1% to the two meta-options.

If you write an argument that breaks down the 'okay outcomes' into lots of distinct categories, without breaking down internal conjuncts and so on, Reality is very impressed with how disjunctive this sounds and allocates more probability.

"Reality" is either the market participants voting percentages of their capital, or Eliezer Yudkowsky choosing how to resolve this market. (Actual reality, not being an agent, doesn't get impressed and allocate probability to prediction markets by being persuaded; but market creators, as judges of observations of reality, can be) And, because these are prediction markets and not polls, it's the resolution probability that matters(for profit).

So, has "reality" been allocating resolution probability to this option, which self-referentially makes it true? Not quite:

It seems like reality has most recently chosen to bid it up to about 1%, and bid 14% of the money they chose to wager at the time. So maybe this option should resolve 1%, representing how impressed reality is with Manifold's reading comprehension.

What is the other meta-option?

You are fooled by at least one option on this list, which out of many tries, ends up sufficiently well-aimed at your personal ideals / prejudices / the parts you understand less well / your own personal indulgences in wishful thinking.

Scrolling through the trade history, that's definitely been happening. It seems like most traders just scroll down the list until their eyes glaze over from boredom and they buy whatever stuck in their mind.

Since the actual AGI disjunctions effectively resolve NA, the other meta-option probably resolves around 1%, and most traders do seem to be doing this, it seems fine to resolve 99% "You are fooled".

Reality also seems to be bidding up some of the AGI options, about 50% of their cost basis. These could be "non-epistemic bets"; or we may need to wait for Manifold to implement "partial resolutions" so that most of the pool can be distributed to the meta-options leaving the rest to the AGI options(and then canceled). Most of reality's payouts are on the meta-options and option D; and reality seems to be bidding "you are fooled" as high as 7% and with the highest overall cost basis of any option.

What does everyone else think? You've collectively put 91,400 mana into the liquidity pool. As rational profit-motivated bettors, you must expect to profit on resolution when you bet(otherwise you're being irrational), but given prevailing interest rates and anthropic complications, I don't see how that's possible for anything except the meta options. And those options do seem to be happening.

bought Ṁ296 of N/A

Nobody here knows enough about neurofeedback to consider it at all, which is the simplest explanation for why it ended up at the literal bottom of the list when in reality it is in the top 5.

@ooe133 Oops, this statement is worded in a non-bayesian manner. Correction: not enough people here know enough about...

bought Ṁ20 of N/A

Is "AGI gains control of the future but is uninterested in killing existing humans or destroying human civilization" option C, none of the above, or a "not okay" outcome?

@MaxMorehead it's pretty close to E, "Whatever strange motivations end up inside an unalignable AGI, or the internal slice through that AGI which codes its successor, they max out at a universe full of cheerful qualia-bearing life and an okay outcome for existing humans."

@MaxMorehead I think not okay, hard to imagine it will be 20% of max score

bought Ṁ20 of N/A

/

/

bought Ṁ10 of N/A

It's baffling to me that people believe A.) is plausible in any reasonable timespan without AGI, brain-computer interfaces and emulation are so far off. I believe it's possible we'll see greater coordination as described, but I'm gonna go with M.). Perhaps it's just not such a clean catch-22 that you already need aligned AGI to align AGI? If that turns out to be the case, then I'll go with B.)

@ImperishableNeet Agreed, I would short A.) if I could.

Can't this be converted into a multi-binary market so I can bet NO on things?

I made a duplicate where you are allowed to short sell options you think are unlikely: https://manifold.markets/ChristopherKing/if-artificial-general-intelligence-669e44ca740e

bought Ṁ50 of N/A

Why would you post this as an image? You made me scroll through Yudkowsky’s anxiety-inducing Twitter timeline to find the source of this in order to find out the context of what he’s talking about.

https://twitter.com/ESYudkowsky/status/1656150555839062017

Spoiler: he’s talking about OpenAI’s attempts to use GPT-4 to interpret and label the neurons in GPT-2.

I'd like to showcase this market—it concerns an important question, has many detailed yet potentially possible options, and has personally changed how I think about which of these answers is worth maximizing the chances of.

@AndrewG I like this as a social media post but as a prediction market I am frustrated by its high chance of resolving n/a (20% is a lot) and Manifold's DPM mechanism.

bought Ṁ1,000 of N/A

@AndrewG Unfortunately, we don't have a great way of subsidizing DPM markets at the moment. For now I've put in M1000 into "You are fooled by at least one option on this list..."; I didn't want to place more lest I shift probabilities too much