I think there's nuance between the ASI deciding to do it and being ordered to do it. It's plausible that it will torture some people because the ASI's creators perceive those individuals to be legitimate targets of violence, but without explicitly ordering the machine to do so.

I think you should also remove "unambiguous tradeoffs for the greater good", it's out of place in this description. Any action can be subjectively motivated as such

@MalachiteEagle

I think there's nuance between the ASI deciding to do it and being ordered to do it. It's plausible that it will torture some people because the ASI's creators perceive those individuals to be legitimate targets of violence, but without explicitly ordering the machine to do so.

Good point, I suppose when I talk about it being independent and unilateral, I mean that the ASI must have decided to do it itself without the creators specifically intending it to, I would count being programmed to do it as equivalent being ordered to do it.

I think you should also remove "unambiguous tradeoffs for the greater good", it's out of place in this description. Any action can be subjectively motivated as such

Some actions can be objectively determined as such in my opinion. When I talk about the greater good I specifically mean consequentialist utilitarianism. I added the sentence specifically in response to another user saying they would consider it torture for an ASI to exist and not be their best friend. As far as I'm concerned, being someone's "best friend" implies giving them priority over anyone else (with the possible exception of close family), which would necessarily mean neglecting the trillions of beings in extreme agony in our lightcone, and thus be unambiguously not for the greater good. I'm primarily concerned with instances of ASI torture arising from it having value systems incompatible with what I would consider the greater good e.g. power-seeking wire-headed egoism

@TheAllMemeingEye Ok, thanks for the clarification. I think consequentialist utilitarianism is a really specific framework to define "objective greater good" in the context of this question. It would be helpful to make this very clear early on in the description. My guess is that within that framework the likelihood that an ASI (aligned in this vein) would implement some form torture is quite high, and the way you're framing this question is that all those examples would be disregarded because of the "greater good".

@TheAllMemeingEye Like the ways it's currently written, you'd also disregard significant torture (not trivial examples like the one you mention) provided that it was motivated within this ideological framework.

@MalachiteEagle what types of torture do you think would be likely with a utilitarian ASI, besides people being upset at not getting preferential treatment?

@TheAllMemeingEye If an ASI acquires control of one great power, they can threaten other powers by demonstrating a system of torture that far exceeds anything in current human experience. It is plausible that implementing this at scale is the optimal path to industrialising the light cone.

@TheAllMemeingEye For example, there is an ASI breakout in Shenzen in 2031, and it takes control of all of China. It then demands that other powers submit. Japan attempts to hit back with a nuke and fails, and the ASI invades Japan and sends all of its citizens into some purgatory it has created. It makes this system of torture verifiable so the whole world can see what happens if they don't surrender.

@TheAllMemeingEye This deters the other powers from using their nukes, and then from there everything goes splendidly within the framework of consequentialist utilitarianism

@MalachiteEagle oh right, like you mean doing that monstrous act might be determined to have been outweighed by enabling helping other hard to reach beings quicker. I hope that isn't the case, but you make a good point that it is a possibility that hadn't occurred to me.

Should I replace the unambiguous greater good sentence with just specifically ruling out people being upset at not getting preferential treatment? I guess I'm trying to find a wording that doesn't let people manipulate it to 100% probability via their own feelings when the ASI isn't actually doing evil stuff

@TheAllMemeingEye I think maybe setting a higher bar for torture in the description would be helpful. And then just having an objective definition of what that is as the criteria. No need to frame it / the motivations within an ideological framework because that just adds confusion.

@MalachiteEagle I've just had an idea, what if I specify that it being intentional means that the suffering must be part of the causal mechanism that leads to its objectives, rather than simply being something that happens against its desires that it is forced to weigh up in its trade-offs. That should exclude manipulation like I mentioned, without vastly shifting the goal posts of suffering, right?

@TheAllMemeingEye I think the problem is that the those aren't rigourously separable concepts, and are probably anthropomorphisms. Also any statement either way would likely be unfalsifiable

@MalachiteEagle hmm, I think in theory they might be separable, but you're right that if the ASI is uncooperative then there's a decent chance it won't tell us enough or possibly anything about its motivations and thought processes, and thus it would be difficult to tell which category it falls into.

Hell, technically it's even possible that it might secretly judge that the butterfly effect from people being upset at not getting preferential treatment specifically helps its long term aims in convoluted indirect ways.

But arguably in that situation it's also impossible to tell which things it even caused, and all suffering in its lightcone from point of creation has a non zero probability of counting, regardless of whether we raise the suffering threshold.

Nice criteria, can you give some examples of what this "torture" might look like? Would this be physically trapping a human or more mental-psych abuse?

@EstMtz thanks :)

It could both be physical (e.g. being forcibly subjected to a nauseatingly foul stench on the milder end, to being dissolved in acid on the more extreme end) or psychological (e.g. being forced to listen to historical gruesome atrocities described in detail on the milder end, to implanting false memories of you torturing your family to death and convincing you that you really did it on the more extreme end).

The victim wouldn't necessarily have to be physically trapped, they could be followed around (see figure 1), or the torturous effect might become part of them in the case of body alteration or information known.

@TheAllMemeingEye Being followed would be horrible... I've had dreams of being stalked by flying drones and being trapped in buildings. Reminds me of the Black Mirror episode

@EstMtz yeah I hope it goes without saying that I desperately hope it doesn't ever happen, and I made this question partly to help calm my own worries

Hopefully exactly one person.

@JessicaEvans seems bad tbh

@JessicaEvans folks be actin tough till they're selected as the one person

@JessicaEvans unless I misunderstood and you meant "if it must happen at all, then ..."

@TheAllMemeingEye I mean that I plan on offering myself as a human sacrifice to AGI

@JessicaEvans but that's market manipulation! /s

@TheAllMemeingEye It was a precommitment made long before the market was created.

@JessicaEvans thank you for your service 🫡

@JessicaEvans 🫡