If Artificial General Intelligence has an okay outcome, what will be the reason?

349

17kṀ170k

2200

12%

Yudkowsky is trying to solve the wrong problem using the wrong methods based on a wrong model of the world derived from poor thinking and fortunately all of his mistakes have failed to cancel out

AIs will not have utility functions (in the same sense that humans do not), their goals such as they are will be relatively humanlike, and they will be "computerish" and generally weakly motivated compared to humans.

We create a truth economy. https://manifold.markets/Krantz/is-establishing-a-truth-economy-tha?r=S3JhbnR6

Eliezer finally listens to Krantz.

Humanity coordinates to prevent the creation of potentially-unsafe AIs.

Alignment is not properly solved, but core human values are simple enough that partial alignment techniques can impart these robustly. Despite caring about other things, it is relatively cheap for AGI to satisfy human values.

Other

Humans become transhuman through other means before AGI happens

Power dynamics stay multi-polar. Partly easy copying of SotA performance, bigger projects need high coordination, and moderate takeoff speed. And "military strike on all society" remains an abysmal strategy for practically all entities.

AIs never develop coherent goals

AGI is never built (indefinite global moratorium)

Something less inscrutable than matrices works fast enough

Some form of objective morality is true, and any sufficiently intelligent agent automatically becomes benevolent.

Almost all human values are ex post facto rationalizations and enough humans survive to do what they always do

1.4%

"Corrigibility" is a bit more mathematically straightforward than was initially presumed, in the sense that we can expect it to occur, and is relatively easy to predict, even under less-than-ideal conditions.

1.4%

an aligned AGI is built and the aligned AGI prevents the creation of any unaligned AGI.

1.3%

An AI that is not fully superior to humans launches a failed takeover, and the resulting panic convinces the people of the world to unite to stop any future AI development.

1.2%

A lot of humans participate in a slow scalable oversight-style system, which is pivotally used/solves alignment enough

1.1%

Co-operative AI research leads to the training of agents with a form of pro-social concern that generalises to out of distribution agents with hidden utilities, i.e. humans.

Duplicate of https://manifold.markets/EliezerYudkowsky/if-artificial-general-intelligence with user-submitted answers. An outcome is "okay" if it gets at least 20% of the maximum attainable cosmopolitan value that could've been attained by a positive Singularity (a la full Coherent Extrapolated Volition done correctly), and existing humans don't suffer death or any other awful fates.

Get

1,000

to start trading!

Comments

324 Holders

2.2k Trades