Background on me: I rate myself at somewhere between 75th and 95th percentile skill in machine learning engineering [amongst employed professionals of that job title]. I rate myself at >95th percentile knowledge/skill in the overlap space of machine learning and neuroscience. Disclaimer: I may be wrong about these estimates, this is just information for you to use about my opinion of myself.
Resolution criteria: I must either have successfully done this, or it must be obvious that someone of my equivalent skill level could accomplish this. Coding a seed AI that recursively self-improves to AGI counts, if the improvement is accomplished within budget and timeframe.
Criteria for AGI pretty vauge, but I will try to resolve honestly in way which I believe would be convincing to most neutral parties as of creation of this question. Does not require capability of manipulating robotics for real world tasks, but does require all cogntive skills (long term planning, theory of mind, contextual awareness, episodic and declarative memory, etc). At the very least this would require demonstrating use of a virtual mouse & keyboard to accomplish complex multistage tasks on a web browser (e.g. open an email account and send emails, download a github repo and run the code therein, rent cloud computing resources and utilize them to train ML models, write a research paper about a previously unknown topic), play any Steam game to at least 90th percentile capability, drive a small RC car with a camera through obstacles or on complex navigation tasks to at least 90th percentile human capability, write a 500 page novel with logically cohesive plot, get at least a 4 on any US highschool AP exam without external resources, score at least 90th percentile vs humans on all existing non-robotic AI benchmarks (avalance continual learning benchmarks, minecraft, BIG bench, text adventure games, etc), score at least 90th percentile vs humans on most Kaggle competitions, convincingly engage in protracted dialogue with adversarial questioners (Turing test).
May include open source code developed by others, including libraries and use of pre-trained open source models.
Related questions
@toms > May include open source code developed by others.
Presumably there is going to need to be iterative experimentation.
P(anyone can train their own AGI | AGI exists by 2030) * P(AGI exists by 2030). I think the second number is around 50%, and the first number is probably less than 30%. I think a world where anyone can train their own AGI (not much different from "Nathan can train their own AGI") is very unstable and we probably wouldn't settle on such a state, or wouldn't last long if we did
@Nikola why would it be hard to train an agi in the presence of friendly superintelligences? we either get the world where basically everyone can do it, or we get the world where we're all dead.
@L I would imagine a singleton-controlled (or controlled by a small set of AGIs, which partition the universe into their respective territories) to be much more likely than a world where anyone can train their own AGI. If anyone can train their own AGI, then we would either need a global surveillance state which prevents misaligned AGIs, or we would die very quickly due to unilateral actors deploying malicious AGI. If the offense-defense balance is merciful enough, this kind of world would be much more sustainable, but in general it seems easier for an individual to destroy things than create them
@Nikola where do you put P(AGI is superintelligent and is generally agreed to control the world or large portions of it | AGI exists by 2030)? because I assign like 2% credence to that.
@Adam I think superintelligence will probably happen like 1 week to 1 year after AGI (depending on whether the main bottleneck is in amount of compute, or in intelligence per unit compute, and on how hard it is to get lots of compute really quickly), so around 60% for P(ASI by 2030 | AGI by 2030). I think superintelligence probably has like a 1/2 chance of taking over the world pretty much instantly if it wanted to, so a very rough estimate is 60% * 1/2 = 30%
@Nikola Thanks Nikola, this is exactly the sort of discussion I hoped to encourage with this question. Personally, I'm more imagining some sort of global surveillance state which stops small AGIs from getting too powerful. I think your view is also quite plausible though. I'm honestly quite uncertain how things are going to work out.
@JonathanSimon anything open-source enough for me to make use of without being part of a specific organization counts.
@NathanHelmBurger Well that certainly makes it easier. I guess the big unknown then will be how easy it is to finetune these models for new tasks. For example I'm sure the human data-labeling + PPO fine-tuning required to turn GPT3-->InstructGPT cost >$10k. And we'd presumably be talking about fine-tuning something much larger (barring a Chinchilla-inspired model downscaling trend). So basically there would need to be one-or-more out-of-the-box models that basically do what you want, or could be made to due them via simple prompting.
@jonsimon Yes, it's more a question of how weird the world is going to be if everyone with moderate resources is capable of making their own small AGI model, fine-tuned to their preferences. What stops them from recursive self-improvement? From independently gathering resources and runaway duplication? The trends of compute and algorithmic improvement point at the destination, but the destination seems nuts. So what happens betwixt now and then? I hope people will share their speculations in the comments.
I think the real question is whether you can do this now. because anyone who's paying attention probably can! since openai have a literal text hlmi, and they're using an old slow architecture to do it, it should only require plugging together the latest fast models with the appropriate multimodal inputs. I think actually this is likely safer than using pretrained models, because if you're doing embodied learning, your new custom ai wouldn't share training data with them, and so wouldn't share failure modes. the question is how to scalably ensure that 1. the ai is honest with self and other, even before words are introduced to the new learning system, and 2. the ai is
obviously nobody who isn't in machine learning will agree with me, and most of the ones who are are going to agree even less. but I'll put up to show I'm serious. I'm not going to provide evidence besides claims, though. if you want to know, dm me on another platform or look it up. it really isn't that hard to make an agi at this point.
@L ... hit send too soon. 2. the ai is careful to be clear about what they know and don't know.
@L I'll ask a question tomorrow about my current preferred ai safety plan, since we're nearing the end of crunch time we really need to get moving on this.
@L You should make some markets capturing your timelines so you can say I told you so if you're right. Or send me a list of questions and I'll make them
@vluzko your markets are all great metrics, and I would buy only buy a little bit of "sooner" on timelines markets for many of them for values not that much sooner than the times you asked about. I also expect to lose some of my yesses, because even with the timelines I expect, I don't think we're going to see every viable capability realized by the times you've asked about. These outcomes would each individually blow me away and severely exceed my timeline expectations:
90% of IMO problems by end of 2023
adam replaced by 2024 (though as I say this, I notice myself quaking a little at some recent papers...)
realistic video from description by end of jan 2023
realistic video or image without strange structural errors by end of 2023
entry level ai coder by start of 2024 (entry level coder is primarily safety bottlenecked, which afaict is the current frontier)
watch a movie and tell you what's going on by end of 2023
explain formal language proofs by start of 2024 (this one is very very hard for humans; stuff that requires a stellar college grad is much longer timelines than mere scaling.)
the thing backing all of this: I expect dramatic scaling from integrating various little known but mostly already-published algorithms. the problem is figuring out exactly which ones; I've worked with some promising ones enough to feel it's likely numenta isn't completely full of crap about their overall research plan, though admittedly I bring them up as the example because they're not one of the more effective research groups in the subfield. I think deepmind also probably has ideas for how to surprise people with scaling.
I'd bet a lot of yes on "will software-side scaling appear to be suddenly discontinuous before 2025?" and a lot of no on "will software-side scaling appear to be suddenly discontinuous before june 2023?", that's approximately my 80% window for when these basic capabilities happen. it might turn out that the software I'm expecting also depends on some hardware changes, though I'd currently bet >50% no new hardware is required and the algorithms will work on a 3090.
@L oh, also, better than humans at everything by 2024 is a bit much - it seems quite plausible to me but would be a bit of a shock. I'm expecting a smoother increase where we use up most of the remaining basic algorithms overhead in both software and hardware in the next 2 years, and then it turns out that raising AI that understands the world deeply just takes a while running on actual robots. but we're going to need to have solved fairly strong safety by that point, because the AI that can be superhuman at everything if scaled and trained through a long slog of sequential life learning is coming very, very soon.
@L Re: "dm me on another platform" Since you've chosen to identify just as 'L' on this platform I can't contact you elsewhere unless you say you would like me to actively attempt to circumvent your anonymity and I succeed at doing so. Feel free to contact me at 'Nathan Helm-Burger' on LessWrong or reach out to me elsewhere.
@L technically, today is before 2030. So if I can do it now, I can do it by 2030. This market isn't about what I believe. I already know that. It's what others believe that I am curious about.