🏅 Top traders

Unless someone persuades me otherwise I’m going to resolve this to 98% on sept 27 because that’s my confidence that an AI can’t communicate faster than the speed of light. The other 2% is based on the conjunction of the possibility that we’re in a simulated universe and the AI will figure out how to hack out of the simulation (if we’re in a simulated universe the simulation was most likely designed by an artificial superintelligence in the outer universe so it won’t be hackable)

ASI can't solve the halting problem or construct a consistent complete axiomatic theory of mathematics. Good chance it can't communicate faster than the speed of light or reduce entropy.

@peterpumpkin but that has nothing to do with ASI... it has everything to do with our current best theories about physics. how are the two related at all? i seriously don't understand what this question is supposed to be answering

@peterpumpkin the question seems to be asking whether humans should be confident period if we have any good limits from physical/math theories regardless of ASI. you could just as well ask if we should be confident in the face of some ET intelligence.. but how on earth would we answer that?

@AdamTreat The question is asking how much our knowledge of physics constrains our expectations of what a superintelligence can physically do. It’s not rocket science, but part of it is literal rocket science haha

“Good chance it can't communicate faster than the speed of light or reduce entropy.” seems sufficient for a yes resolution

I think we can be very, very confident in those claims.

I agree the question doesn't have anything to do with ASI specifically. It's an interesting question though what constraints we can make on future technology. Like how confident are we on light speed limits? Most scifi assumes FTL travel is possible. What about conservation of energy?

@peterpumpkin you can arguably make an assertion that the more often a trope is used in sci-fi, then the less confident we are?

@peterpumpkin What about local, brief violations of various thermodynamic laws to do things that are prohibited on average? i.e. many quantum systems can violate the 2nd law in specific instances, but never on average.

Ribosomes can only construct arbitrary 1-dimensional sequences of a limited set of amino acids. Atomically precise manufacturing in the most general 3d sense is probably impossible.

@JonathanRay

Not sure that’s necessarily true. Could we do atomically precise manufacturing at mK (or microKelvin?) temperatures to get around atoms dodging around too much? Ribosomes operate in much hotter, noisier environments.

@QuantumObserver Creating the necessary chemical bonds releases too much energy. One stray photon from that could mess it up.

@JonathanRay Surely the fact that we can create ultra high purity silicon crystals suggests that is not always the case?

@QuantumObserver In that case you don’t care about atoms going into the wrong slots because all the atoms are silicon

What is ASI?

@AdamTreat ChatGPT tells me in the context of this question it means, "If it is "Artificial Superintelligence."

If that is so, I think the concept is so vague that it is impossible to answer. If you forced me I'd say that physics has nothing to do with it as far as our human conception of physics is defined.

For instance, it is I think unarguable that we right now have ASI in specific domains: chess, protein folding, etc and physics is giving us zero insight into just how much ELO stockfish might gain nor does it give us any insight into what proteins Alpha might discover can be folded/how.

Good question! I think a good way to think about it is trying to imagine which groups could decently infer the limits of our modern technology, and then try to decide which group we're most analogous to in comparison to ASI.

• Could the smartest ancient Egyptians infer the limits of modern technology? (I think 10-20%)

• Could the smartest ancient Athenian philosophers?

• Enlightenment scholars?

• Industrial Revolution scholars?

• The best minds at the manhattan project? (I think 80-90%)

Back to the original question, I think we're somewhere above 50%, but not sure if my scale agrees with you.

@Joshua

I think the answer for any pre-industrial thinkers is basically 0. Why? Because until Sadi Carnot wrote his book, many experts in the design, manufacture, and operation of steam engines thought it was possible to get any amount of work out of some amount of coal, as long as you were clever enough.

Without knowing anything about thermodynamics or electromagnetism, it’s hard to imagine what limits an industrial society must have.

@Joshua The question is asking for “some limits”. Not a limit in every domain, or in every detail, just some upper bounds in some specific domains.

Hmmmm yeah I'm trying to imagine specifically what like a Greek Philosopher would have said if we asked them what it would be definitely impossible for man to do in a few thousand years.

I feel like putting aside their mythology, they might have gone for something like "Man won't be able to move the sun" and been right? Maybe I'm giving them too much credit though.

@Joshua They might have said we won’t be able to resurrect Socrates after his body has decayed

Getting back to the main question of ASI limits... I think we at least get a few dozen percent from fermi paradox math?

We expect there to be other life out there somewhere, and so far it hasn't done anything notable to the universe we can observe. I think that backs up our understanding of physics enough that we can be pretty confident about limits like FTL travel.