@Mira How do I create a group and add markets to it?

@ShadowyZephyr link

@MP 50/50 on whether we have AIs and 50/50 on whether they'll wipe us out. Simples.

@eclair4151 thats losing 70k just if anyone's curious

@8 If AI kills us all, the mana is worthless anyway, so really you can only win

@deagol nice.

@deagol good to see markets are behaving rationally and metamarkets don't massively distort incentives...

@firstuserhere But who cares for long term markets? Here's one for the next 6 months:

@firstuserhere also a duplicate

@MartinRandall yeah, realized later, created the market for this month if you're interested in that

@StrayClimb Uploaded humans count, this is humanity, not biological humanity.

@MartinRandall what if all surviving humans are preserved for bio research? or as tech slaves, say, doing backups and training the AI through captchas nonstop 10h/day?

@deagol Then we would not be wiped out, and I will be sure to be a good little slave and always rotate the backup disks on the right schedule. Hopefully I will have some spare time to resolve this market but if not I'm sure the benevolent AI masters will take care of it.

@deagol maybe

/Zardoru/will-ai-take-over-the-world-and-kee

Is of interest to you?

@MartinRandall not much humanity left in that, but ok gotcha

@deagol Some of our ancestors had pretty miserable lives, but it would be a huge downgrade.

@MartinRandall right. was just confirming whether your word choice of “humanity” just meant humans alive (or uploaded as sentient minds), or perhaps it was something more subjective.

@duck_master

@duck_master oops

@deagol that -3k profit came from the ai letter market

@8 a bit more?

@deagol haha yeah but i was at 19k profit before

@8 and a billion mana before that in cash

@Irigi Also, the risk per year gets very small after 2030.

@Irigi you might want to graph it and use liquidity to infer error bars. Some markets have less liquidity and thus larger error bars.

@MartinRandall the 2030, 2040 and 2060 markets have a large number of investors, and small error bars. From these you can conclude that the risk per year gets very small after 2030.

@TeddyWeverka agreed, I was not responding to the possible belief inconsistencies.

Can't quite take the numbers at face value due to discount rates and relative value of mana and such, but still very interesting.

@Irigi Btw. I am more wary about scenarios like "we will willingly surround ourselves by intelligent machines until it is clear they have lot of real power in hardware and numbers", in particularly in willing misuse, compared to this "very quick doom by superintelligence magic" scenario I was assessing here.

@Irigi Nice try AGI. Using fancy probability estimations to persuade us to push down no and increase the ods, just so that you can bet yes before unleashing the nanobot swarm.

@Irigi

p_s is 20%? I think this is above 99%. I frequently interact with people who are much smarter than me in their area of expertise. They just win. And this is purely within the human realm of intelligence, and purely a 1:1 competition.

Chaotic systems are harder to predict but conversely easier to control. Overall chaotic systems are an advantage to a force with more intelligence and lower initial resources.

@MartinRandall Thank you, it is good to discuss the numbers! Are you assigning probability to "machine can be smarter than human" or to "machine can be so much smarter we cannot win (in surviving) no matter what we do"? In the first one I am with you, surely there are much smarter people who win already in the human intelligence span. And in particular, there are fields that are very deep and unsaturated (e.g. many fields of science), and if the competition is fair. But for the second one I am surprised that you put 99% on something that we never saw instance of and we only expect by extrapolation. (My 20% probably is too low, maybe it should be closer to 50%, with quite wide distribution - in the end, these product equations are better treated as distributions rather than mean values similar to arxiv.org/abs/1806.02404.)

> Chaotic systems are harder to predict but conversely easier to control.
Do you have some example / reference? My experience / assumption is that they are uncontrollable in praxis and therefore serve as 'planning horizon'. By analogy, if in chess after each move one piece was randomly shifted, it would probably hinder planning. Particularly for very strong players. You could still play the game, but while the rating ladder now spans something like 800-3500 and is probably close to saturation due to draw-death of chess, the span of the "chess with randomness" would be much narrower.

@Irigi for example, tides are very linear but very hard to change. Whereas the weather is chaotic and easier to change. A double pendulum is very chaotic but modern robots can balance them easily. Humans are too slow, of course.

A feature of chaotic systems is that very small changes can cause very large effects. This makes them easier to control than linear systems, where very large effects require very large changes.

Truly random systems are harder to predict and to control, but you were talking about chaotic systems, not random ones. We could discuss randomness separately but I doubt you think that true randomness is a factor in your probability.

@Irigi regarding "machine can be so much smarter we cannot win (in surviving) no matter what we do", if that is your definition of p_s, what are p_1 and p_f doing in your model?

My 99%+ is for "we would be to superintelligence as animals / 3 years olds are to us", which is your definition of p_s. Given sufficient advantages my 3yo can beat me at Candyland, yes. Not really relevant for predicting power dynamics.

@MartinRandall The double pendulum is nice example that if you know the dynamics, and can observe the macroscopic degrees of freedom well, and can apply sufficient forcing, you can control the system, probably with much less forcing than linear one.

Can you imagine the same for e.g. weather? I would expect it is hard to apply sufficient forcing to the macroscopic degrees of freedom, as well as observe them precisely. (In the end, the target system is probably economy or society, but I do not have good estimate how chaotic are these).

If the chaotic system cannot be controlled, you are left with "prediction horizon", whether due to true randomness (random quantum effects amplified by the chaotic system?) or just unknown state of the full system.

@MartinRandall

>"machine can be so much smarter we cannot win (in surviving) no matter what we do", if that is your definition of p_s, what are p_1 and p_f doing in your model?

Ok, point taken. 99% still seems way too high, but p_s definitely should be much higher than 20%.

@Irigi sure, so now we move on to p_1. This is supposed to be the probability that given that "we would be to superintelligence as animals / 3 years olds are to us", and also that "capabilities will be so fast that we will not be able to respond in time by a proper policy", given those things as already true, "the first attempt to get rid of humanity is successful".

Well, suppose that I am trying to take candy from a baby. And further suppose that the baby is not able to respond in time. Will my first attempt to take the candy be successful?

I don't think this is 99% only because there may be race dynamics with other AIs that are also trying to take over the world, in the scenario where lethal AGI is deployed in parallel.

I take such attempts that we would probably try to shut AI down afterwards

Conditional on humanity deploying a lethal AI and miraculously surviving, the odds that we would be exactly that stupid again seem pretty high to me. Plus the odds of multiple AIs trying to take over the world at once. Plus the odds that the ensuing chaos causes a death spiral.

Generally when modeling risk you should be adding up all the known and unknown pathways to risk, not estimating the odds of a single pathway. Your estimate will be too low otherwise. And I say this as a NO better who agrees than 21% by 2030 is too high.

@MartinRandall
> Well, suppose that I am trying to take candy from a baby. And further suppose that the baby is not able to respond in time. Will my first attempt to take the candy be successful?

There is the problem with analogies, they do not contain all the information. For example, this one mixes intelligence, power and how much the entity is trusting. Suppose you are a man with no special equipment and you would like to hunt a bear. You also know that it is deaf and will not be able to respond in time. Will you succeed? Perhaps you will make a improvised weapon and hurt it in the first attack, and then it will kill you, or run away. Or you conclude it is not a good idea to do in the first place. If civilization equiped you with a gun, you would probably win - but we would not be facing civilization of AGIs, just the first instances.

The bear is probably not more intelligent than 3 years old, but is more powerful and definitely not trusting. If it would be a bear with the nature of a child, you could trick it in many ways and it would most likely not use his power.

I am not saying we are in the man-bear situation, but we are not in the man-child situation either. Moreover, some components of humanity are trusting, and some are paranoid and powerful.

> Conditional on humanity deploying a lethal AI and miraculously surviving, the odds that we would be exactly that stupid again seem pretty high to me.
Probably right, but I would put nuclear weapons as a counter example. (But then, nuclear weapons do not spit gold, if I borrow this analogy).

@DylanSlagh What are your current odds of AI wiping out humanity by 2030, ignoring that factor?

@MartinRandall Maybe something like 10%. Even though that seems like a big percentage it feels very difficult to act on that belief. Most of the AI doom discourse I read is extremely low quality I find almost everyone’s tone to be insufferable on both sides

@DylanSlagh It is hard to act differently, but broadly we can up-weight the importance of the next 6-7 years by 10%. The personal difference between AI doom and nuclear doom is small. Of course the civilizational difference is larger.

And of course we can bet NO in this market as long as it is at 20%.