I assume this resolves no if you're dead.

@MartinRandall Correct.

Relevant:

@PatrickDelaney How?

@toms > May include open source code developed by others.

Presumably there is going to need to be iterative experimentation.

Llama = $1m, maybe $300k for a baby llama

Compute ~30x cheaper in 7.5yrs,

Software improves slightly faster than compute

Baby llama equivalent ~$300,

Add planning, vision, etc. x 10-30

AGI = $10k

(clearly motor tasks are an order of magnitude harder, cf cerebellum/cerebrum synapse counts, hence the typical ~2035 for laptop = human brain matches this as well)

P(anyone can train their own AGI | AGI exists by 2030) * P(AGI exists by 2030). I think the second number is around 50%, and the first number is probably less than 30%. I think a world where anyone can train their own AGI (not much different from "Nathan can train their own AGI") is very unstable and we probably wouldn't settle on such a state, or wouldn't last long if we did

@Nikola why would it be hard to train an agi in the presence of friendly superintelligences? we either get the world where basically everyone can do it, or we get the world where we're all dead.

@L I would imagine a singleton-controlled (or controlled by a small set of AGIs, which partition the universe into their respective territories) to be much more likely than a world where anyone can train their own AGI. If anyone can train their own AGI, then we would either need a global surveillance state which prevents misaligned AGIs, or we would die very quickly due to unilateral actors deploying malicious AGI. If the offense-defense balance is merciful enough, this kind of world would be much more sustainable, but in general it seems easier for an individual to destroy things than create them

@Nikola where do you put P(AGI is superintelligent and is generally agreed to control the world or large portions of it | AGI exists by 2030)? because I assign like 2% credence to that.

@Adam I think superintelligence will probably happen like 1 week to 1 year after AGI (depending on whether the main bottleneck is in amount of compute, or in intelligence per unit compute, and on how hard it is to get lots of compute really quickly), so around 60% for P(ASI by 2030 | AGI by 2030). I think superintelligence probably has like a 1/2 chance of taking over the world pretty much instantly if it wanted to, so a very rough estimate is 60% * 1/2 = 30%

@Nikola Thanks Nikola, this is exactly the sort of discussion I hoped to encourage with this question. Personally, I'm more imagining some sort of global surveillance state which stops small AGIs from getting too powerful. I think your view is also quite plausible though. I'm honestly quite uncertain how things are going to work out.

Do you have to be able to train such a system from scratch for $10k, or does this still resolve to yes if you do some sort of trivial fine-tuning on top of an existing ~$10M model?

@JonathanSimon anything open-source enough for me to make use of without being part of a specific organization counts.

@NathanHelmBurger Well that certainly makes it easier. I guess the big unknown then will be how easy it is to finetune these models for new tasks. For example I'm sure the human data-labeling + PPO fine-tuning required to turn GPT3-->InstructGPT cost >$10k. And we'd presumably be talking about fine-tuning something much larger (barring a Chinchilla-inspired model downscaling trend). So basically there would need to be one-or-more out-of-the-box models that basically do what you want, or could be made to due them via simple prompting.

@jonsimon Yes, it's more a question of how weird the world is going to be if everyone with moderate resources is capable of making their own small AGI model, fine-tuned to their preferences. What stops them from recursive self-improvement? From independently gathering resources and runaway duplication? The trends of compute and algorithmic improvement point at the destination, but the destination seems nuts. So what happens betwixt now and then? I hope people will share their speculations in the comments.

I think the real question is whether you can do this now. because anyone who's paying attention probably can! since openai have a literal text hlmi, and they're using an old slow architecture to do it, it should only require plugging together the latest fast models with the appropriate multimodal inputs. I think actually this is likely safer than using pretrained models, because if you're doing embodied learning, your new custom ai wouldn't share training data with them, and so wouldn't share failure modes. the question is how to scalably ensure that 1. the ai is honest with self and other, even before words are introduced to the new learning system, and 2. the ai is

obviously nobody who isn't in machine learning will agree with me, and most of the ones who are are going to agree even less. but I'll put up to show I'm serious. I'm not going to provide evidence besides claims, though. if you want to know, dm me on another platform or look it up. it really isn't that hard to make an agi at this point.

@L ... hit send too soon. 2. the ai is careful to be clear about what they know and don't know.

@L I'll ask a question tomorrow about my current preferred ai safety plan, since we're nearing the end of crunch time we really need to get moving on this.

@L You should make some markets capturing your timelines so you can say I told you so if you're right. Or send me a list of questions and I'll make them

@vluzko your markets are all great metrics, and I would buy only buy a little bit of "sooner" on timelines markets for many of them for values not that much sooner than the times you asked about. I also expect to lose some of my yesses, because even with the timelines I expect, I don't think we're going to see every viable capability realized by the times you've asked about. These outcomes would each individually blow me away and severely exceed my timeline expectations:

• 90% of IMO problems by end of 2023

• adam replaced by 2024 (though as I say this, I notice myself quaking a little at some recent papers...)

• realistic video from description by end of jan 2023

• realistic video or image without strange structural errors by end of 2023

• entry level ai coder by start of 2024 (entry level coder is primarily safety bottlenecked, which afaict is the current frontier)

• watch a movie and tell you what's going on by end of 2023

• explain formal language proofs by start of 2024 (this one is very very hard for humans; stuff that requires a stellar college grad is much longer timelines than mere scaling.)

the thing backing all of this: I expect dramatic scaling from integrating various little known but mostly already-published algorithms. the problem is figuring out exactly which ones; I've worked with some promising ones enough to feel it's likely numenta isn't completely full of crap about their overall research plan, though admittedly I bring them up as the example because they're not one of the more effective research groups in the subfield. I think deepmind also probably has ideas for how to surprise people with scaling.

I'd bet a lot of yes on "will software-side scaling appear to be suddenly discontinuous before 2025?" and a lot of no on "will software-side scaling appear to be suddenly discontinuous before june 2023?", that's approximately my 80% window for when these basic capabilities happen. it might turn out that the software I'm expecting also depends on some hardware changes, though I'd currently bet >50% no new hardware is required and the algorithms will work on a 3090.

@L oh, also, better than humans at everything by 2024 is a bit much - it seems quite plausible to me but would be a bit of a shock. I'm expecting a smoother increase where we use up most of the remaining basic algorithms overhead in both software and hardware in the next 2 years, and then it turns out that raising AI that understands the world deeply just takes a while running on actual robots. but we're going to need to have solved fairly strong safety by that point, because the AI that can be superhuman at everything if scaled and trained through a long slog of sequential life learning is coming very, very soon.

@L Re: "dm me on another platform" Since you've chosen to identify just as 'L' on this platform I can't contact you elsewhere unless you say you would like me to actively attempt to circumvent your anonymity and I succeed at doing so. Feel free to contact me at 'Nathan Helm-Burger' on LessWrong or reach out to me elsewhere.

@L https://manifold.markets/vluzko/will-softwareside-ai-scaling-appear?referrer=ian

@L technically, today is before 2030. So if I can do it now, I can do it by 2030. This market isn't about what I believe. I already know that. It's what others believe that I am curious about.