I am an AI safety researcher with a background in machine learning engineering and neuroscience. Will I personally be able to program and train an AGI for less than $10k by 2030?
22%
chance

Background on me: I rate myself at somewhere between 75th and 95th percentile skill in machine learning engineering [amongst employed professionals of that job title]. I rate myself at >95th percentile knowledge/skill in the overlap space of machine learning and neuroscience. Disclaimer: I may be wrong about these estimates, this is just information for you to use about my opinion of myself.

Resolution criteria: I must either have successfully done this, or it must be obvious that someone of my equivalent skill level could accomplish this. Coding a seed AI that recursively self-improves to AGI counts, if the improvement is accomplished within budget and timeframe.

Criteria for AGI pretty vauge, but I will try to resolve honestly in way which I believe would be convincing to most neutral parties as of creation of this question. Does not require capability of manipulating robotics for real world tasks, but does require all cogntive skills (long term planning, theory of mind, contextual awareness, episodic and declarative memory, etc). At the very least this would require demonstrating use of a virtual mouse & keyboard to accomplish complex multistage tasks on a web browser (e.g. open an email account and send emails, download a github repo and run the code therein, rent cloud computing resources and utilize them to train ML models, write a research paper about a previously unknown topic), play any Steam game to at least 90th percentile capability, drive a small RC car with a camera through obstacles or on complex navigation tasks to at least 90th percentile human capability, write a 500 page novel with logically cohesive plot, get at least a 4 on any US highschool AP exam without external resources, score at least 90th percentile vs humans on all existing non-robotic AI benchmarks (avalance continual learning benchmarks, minecraft, BIG bench, text adventure games, etc), score at least 90th percentile vs humans on most Kaggle competitions, convincingly engage in protracted dialogue with adversarial questioners (Turing test).

May include open source code developed by others, including libraries and use of pre-trained open source models.

Sort by:
Nikola avatar
Nikolabought Ṁ10 of NO

P(anyone can train their own AGI | AGI exists by 2030) * P(AGI exists by 2030). I think the second number is around 50%, and the first number is probably less than 30%. I think a world where anyone can train their own AGI (not much different from "Nathan can train their own AGI") is very unstable and we probably wouldn't settle on such a state, or wouldn't last long if we did

L avatar
Lis predicting YES at 26%

@Nikola why would it be hard to train an agi in the presence of friendly superintelligences? we either get the world where basically everyone can do it, or we get the world where we're all dead.

Nikola avatar
Nikolais predicting NO at 26%

@L I would imagine a singleton-controlled (or controlled by a small set of AGIs, which partition the universe into their respective territories) to be much more likely than a world where anyone can train their own AGI. If anyone can train their own AGI, then we would either need a global surveillance state which prevents misaligned AGIs, or we would die very quickly due to unilateral actors deploying malicious AGI. If the offense-defense balance is merciful enough, this kind of world would be much more sustainable, but in general it seems easier for an individual to destroy things than create them

Adam avatar
Adamis predicting NO at 26%

@Nikola where do you put P(AGI is superintelligent and is generally agreed to control the world or large portions of it | AGI exists by 2030)? because I assign like 2% credence to that.

Nikola avatar
Nikolais predicting NO at 26%

@Adam I think superintelligence will probably happen like 1 week to 1 year after AGI (depending on whether the main bottleneck is in amount of compute, or in intelligence per unit compute, and on how hard it is to get lots of compute really quickly), so around 60% for P(ASI by 2030 | AGI by 2030). I think superintelligence probably has like a 1/2 chance of taking over the world pretty much instantly if it wanted to, so a very rough estimate is 60% * 1/2 = 30%

jonsimon avatar
Jon Simon

Do you have to be able to train such a system from scratch for $10k, or does this still resolve to yes if you do some sort of trivial fine-tuning on top of an existing ~$10M model?

NathanHelmBurger avatar
Nathan

@JonathanSimon anything open-source enough for me to make use of without being part of a specific organization counts.

jonsimon avatar
Jon Simon

@NathanHelmBurger Well that certainly makes it easier. I guess the big unknown then will be how easy it is to finetune these models for new tasks. For example I'm sure the human data-labeling + PPO fine-tuning required to turn GPT3-->InstructGPT cost >$10k. And we'd presumably be talking about fine-tuning something much larger (barring a Chinchilla-inspired model downscaling trend). So basically there would need to be one-or-more out-of-the-box models that basically do what you want, or could be made to due them via simple prompting.

L avatar
Lis predicting YES at 22%

I think the real question is whether you can do this now. because anyone who's paying attention probably can! since openai have a literal text hlmi, and they're using an old slow architecture to do it, it should only require plugging together the latest fast models with the appropriate multimodal inputs. I think actually this is likely safer than using pretrained models, because if you're doing embodied learning, your new custom ai wouldn't share training data with them, and so wouldn't share failure modes. the question is how to scalably ensure that 1. the ai is honest with self and other, even before words are introduced to the new learning system, and 2. the ai is

obviously nobody who isn't in machine learning will agree with me, and most of the ones who are are going to agree even less. but I'll put up to show I'm serious. I'm not going to provide evidence besides claims, though. if you want to know, dm me on another platform or look it up. it really isn't that hard to make an agi at this point.

L avatar
Lis predicting YES at 22%

@L ... hit send too soon. 2. the ai is careful to be clear about what they know and don't know.

L avatar
Lis predicting YES at 22%

@L I'll ask a question tomorrow about my current preferred ai safety plan, since we're nearing the end of crunch time we really need to get moving on this.

vluzko avatar
Vincent Luczkowbought Ṁ30 of NO

@L You should make some markets capturing your timelines so you can say I told you so if you're right. Or send me a list of questions and I'll make them

L avatar
Lis predicting YES at 27%

@vluzko your markets are all great metrics, and I would buy only buy a little bit of "sooner" on timelines markets for many of them for values not that much sooner than the times you asked about. I also expect to lose some of my yesses, because even with the timelines I expect, I don't think we're going to see every viable capability realized by the times you've asked about. These outcomes would each individually blow me away and severely exceed my timeline expectations:

  • 90% of IMO problems by end of 2023

  • adam replaced by 2024 (though as I say this, I notice myself quaking a little at some recent papers...)

  • realistic video from description by end of jan 2023

  • realistic video or image without strange structural errors by end of 2023

  • entry level ai coder by start of 2024 (entry level coder is primarily safety bottlenecked, which afaict is the current frontier)

  • watch a movie and tell you what's going on by end of 2023

  • explain formal language proofs by start of 2024 (this one is very very hard for humans; stuff that requires a stellar college grad is much longer timelines than mere scaling.)

the thing backing all of this: I expect dramatic scaling from integrating various little known but mostly already-published algorithms. the problem is figuring out exactly which ones; I've worked with some promising ones enough to feel it's likely numenta isn't completely full of crap about their overall research plan, though admittedly I bring them up as the example because they're not one of the more effective research groups in the subfield. I think deepmind also probably has ideas for how to surprise people with scaling.

I'd bet a lot of yes on "will software-side scaling appear to be suddenly discontinuous before 2025?" and a lot of no on "will software-side scaling appear to be suddenly discontinuous before june 2023?", that's approximately my 80% window for when these basic capabilities happen. it might turn out that the software I'm expecting also depends on some hardware changes, though I'd currently bet >50% no new hardware is required and the algorithms will work on a 3090.

L avatar
Lis predicting YES at 23%

@L oh, also, better than humans at everything by 2024 is a bit much - it seems quite plausible to me but would be a bit of a shock. I'm expecting a smoother increase where we use up most of the remaining basic algorithms overhead in both software and hardware in the next 2 years, and then it turns out that raising AI that understands the world deeply just takes a while running on actual robots. but we're going to need to have solved fairly strong safety by that point, because the AI that can be superhuman at everything if scaled and trained through a long slog of sequential life learning is coming very, very soon.

NathanHelmBurger avatar
Nathan

@L Re: "dm me on another platform" Since you've chosen to identify just as 'L' on this platform I can't contact you elsewhere unless you say you would like me to actively attempt to circumvent your anonymity and I succeed at doing so. Feel free to contact me at 'Nathan Helm-Burger' on LessWrong or reach out to me elsewhere.

NathanHelmBurger avatar
Nathan

@L technically, today is before 2030. So if I can do it now, I can do it by 2030. This market isn't about what I believe. I already know that. It's what others believe that I am curious about.

L avatar
Lis predicting YES at 22%

this will just be seen as a form of having a kid by then

L avatar
Lis predicting YES at 22%

@L it is going to get extremely easy to do

L avatar
Lis predicting YES at 22%

@L memetics-only children versus genetics-included children

Adam avatar
Adambought Ṁ100 of NO

"I rate myself at somewhere between 75th and 95th percentile skill in machine learning engineering." wait, so at least 1 in 20 people are better at machine learning engineering than you? you must be a really bad AI safety researcher 😉

NathanHelmBurger avatar
Nathan

@Adam lol, amongst employed professionals of that job title. I figured that was the implied reference class...