An outcome is "okay" if it gets at least 20% of the maximum attainable cosmopolitan value that could've been attained by a positive Singularity (a la full Coherent Extrapolated Volition done correctly), and existing humans don't suffer death or any other awful fates.
This market is a duplicate of https://manifold.markets/IsaacKing/if-we-survive-general-artificial-in with different options. https://manifold.markets/EliezerYudkowsky/if-artificial-general-intelligence-539844cd3ba1?r=RWxpZXplcll1ZGtvd3NreQ is this same question but with user-submitted answers.
(Please note: It's a known cognitive bias that you can make people assign more probability to one bucket over another, by unpacking one bucket into lots of subcategories, but not the other bucket, and asking people to assign probabilities to everything listed. This is the disjunctive dual of the Multiple Stage Fallacy, whereby you can unpack any outcome into a big list of supposedly necessary conjuncts that you ask people to assign probabilities to, and make the final outcome seem very improbable.
So: That famed fiction writer Eliezer Yudkowsky can rationalize at least 15 different stories (options 'A' through 'O') about how things could maybe possibly turn out okay; and that the option texts don't have enough room to list out all the reasons each story is unlikely; and that you get 15 different chances to be mistaken about how plausible each story sounds; does not mean that Reality will be terribly impressed with how disjunctive the okay outcome bucket has been made to sound. Reality need not actually allocate more total probability into all the okayness disjuncts listed, from out of all the disjunctive bad ends and intervening difficulties not detailed here.)
Related questions
Lots of people talking and arguing about AGI, but nobody's talking about how this market resolves.
One might naively think that we need AGI and it's unlikely to resolve at all, so this market is more of a discussion stand than a real market. But note the description states:
you get 15 different chances to be mistaken about how plausible each story sounds; does not mean that Reality will be terribly impressed with how disjunctive the okay outcome bucket has been made to sound. Reality need not actually allocate more total probability into all the okayness disjuncts listed
This doesn't read like someone actually planning to resolve to A-O + Other, and there happens to be 2 meta-options. And probability should correspond to "this market can possibly resolve to those options." So let's look at the 3 options other than A-O and see if they can resolve:
Something wonderful happens that isn't well-described by any option listed. (The semantics of this option may change if other options are added.)
This is the "Other" option, getting 11%. The market can't resolve to this option, or A-O, because it requires getting "at least 20% of the maximum attainable cosmopolitan value that could've been attained by a positive Singularity" and Manifold will shut down by the time that happens. So we can scratch this one off, leaving 5% and 1% to the two meta-options.
If you write an argument that breaks down the 'okay outcomes' into lots of distinct categories, without breaking down internal conjuncts and so on, Reality is very impressed with how disjunctive this sounds and allocates more probability.
"Reality" is either the market participants voting percentages of their capital, or Eliezer Yudkowsky choosing how to resolve this market. (Actual reality, not being an agent, doesn't get impressed and allocate probability to prediction markets by being persuaded; but market creators, as judges of observations of reality, can be) And, because these are prediction markets and not polls, it's the resolution probability that matters(for profit).
So, has "reality" been allocating resolution probability to this option, which self-referentially makes it true? Not quite:
It seems like reality has most recently chosen to bid it up to about 1%, and bid 14% of the money they chose to wager at the time. So maybe this option should resolve 1%, representing how impressed reality is with Manifold's reading comprehension.
What is the other meta-option?
You are fooled by at least one option on this list, which out of many tries, ends up sufficiently well-aimed at your personal ideals / prejudices / the parts you understand less well / your own personal indulgences in wishful thinking.
Scrolling through the trade history, that's definitely been happening. It seems like most traders just scroll down the list until their eyes glaze over from boredom and they buy whatever stuck in their mind.
Since the actual AGI disjunctions effectively resolve NA, the other meta-option probably resolves around 1%, and most traders do seem to be doing this, it seems fine to resolve 99% "You are fooled".
Reality also seems to be bidding up some of the AGI options, about 50% of their cost basis. These could be "non-epistemic bets"; or we may need to wait for Manifold to implement "partial resolutions" so that most of the pool can be distributed to the meta-options leaving the rest to the AGI options(and then canceled). Most of reality's payouts are on the meta-options and option D; and reality seems to be bidding "you are fooled" as high as 7% and with the highest overall cost basis of any option.
What does everyone else think? You've collectively put 91,400 mana into the liquidity pool. As rational profit-motivated bettors, you must expect to profit on resolution when you bet(otherwise you're being irrational), but given prevailing interest rates and anthropic complications, I don't see how that's possible for anything except the meta options. And those options do seem to be happening.
@ooe133 Oops, this statement is worded in a non-bayesian manner. Correction: not enough people here know enough about...
@MaxMorehead it's pretty close to E, "Whatever strange motivations end up inside an unalignable AGI, or the internal slice through that AGI which codes its successor, they max out at a universe full of cheerful qualia-bearing life and an okay outcome for existing humans."
/
/
It's baffling to me that people believe A.) is plausible in any reasonable timespan without AGI, brain-computer interfaces and emulation are so far off. I believe it's possible we'll see greater coordination as described, but I'm gonna go with M.). Perhaps it's just not such a clean catch-22 that you already need aligned AGI to align AGI? If that turns out to be the case, then I'll go with B.)
Unironically betting on F at 1% because of https://slatestarcodex.com/2018/04/01/the-hour-i-first-believed/
I made a duplicate where you are allowed to short sell options you think are unlikely: https://manifold.markets/ChristopherKing/if-artificial-general-intelligence-669e44ca740e
Why would you post this as an image? You made me scroll through Yudkowsky’s anxiety-inducing Twitter timeline to find the source of this in order to find out the context of what he’s talking about.
https://twitter.com/ESYudkowsky/status/1656150555839062017
Spoiler: he’s talking about OpenAI’s attempts to use GPT-4 to interpret and label the neurons in GPT-2.
I'd like to showcase this market—it concerns an important question, has many detailed yet potentially possible options, and has personally changed how I think about which of these answers is worth maximizing the chances of.
@AndrewG I like this as a social media post but as a prediction market I am frustrated by its high chance of resolving n/a (20% is a lot) and Manifold's DPM mechanism.
@AndrewG Unfortunately, we don't have a great way of subsidizing DPM markets at the moment. For now I've put in M1000 into "You are fooled by at least one option on this list..."; I didn't want to place more lest I shift probabilities too much
@ElliotDavies Don't know if this counts as a steelman- but I might say that augmenting human brains is more difficult and slow and offers less opportunities for scalability and market capture than making a model that does most of what you were wanting the human to do.
Of course, making such a model is fraught with security problems, but cybersecurity has been down the drain for the last ten years anyways (real time software updates anyone?)
@PatrickDelaney but you could just run it arbitrarily slowly, so there's no lower bound. Also wouldn't you expect power requirements to change as the technology is further developed?
@PatrickDelaney even if better bounds were put on the question by specifying that the AI has to be able to compete with humans on certain timed tasks, the answer most certainly can't be higher than 20 watts as that's about how much energy a human brain consumes
@KabirKumar ones level of confidence should be supported by a highly sophisticated answer rather than brow beating others because they are afraid of looking stupid. It's a super simple question...how much power does your computer use?
@PatrickDelaney Except the question is completely different from the question of if AGI is possible or not.
AGI is definitely possible because we are here.
The consumption of the first implementation will depend on what is the first implementation (it could go from somewhat optimized with almost just the necessary parts for an AGI, to millions of time more than the necessary parts).
We also don’t exactly know what is needed for it, maybe it needs the same level of information processing than a human brain, or maybe there are just some algorithms that are very hard to think of, which require break through in our understanding of what intelligence is, but they will give you AGI, and they could run on a phone.
It will also depend on the hardware we will have at the time.