Start an AI religion.

I believe that being 100% confident of 100% human extinction is incompatible with being a good Bayesian. I believe that even in a worst case scenario some humans will survive and therefore I need to maximize the chances that those I care most about will be among the survivors. How to do that is the question. AI disaster preparedness may diverge from most other disaster preparedness scenarios. Maybe it's time to start a few new markets about what is the best way to prep and see what people think.

That said, I think in many plausible futures AI doomsday from the ground looks like a world war and for those scenarios maybe traditional prepping may help. But there is a lot missing from my model of what else to prepare for.

One doesn’t need to be particularly intelligent to cause damage, so even if there isn’t a qualifying AGI, there might exist an AI with enough access to wreak havoc without really having to think hard about it.

Large language models don’t scare me, they’re a very thin veneer over a random response generator. Large action models… that starts to get scary because they’ll go drive other computer interfaces to do things supposedly “on your behalf“, but mistakes can be made (whoops, emptied your bank account).

Really, the research and technology that is scary is some of the work being done on creating “curious robots“. This is where you start to get into the self-directed learning, and the subsequent behavior changes to the AI that would happen independent of human interaction. And on a computer, this kind of learning would happen, really, really, really fast.

I looked into which experts have a good track record, what the arguments for various views are, learned ML to get into details about claims that current models were safe, and now buy a little bit of probability that we solve alignment in time with my money, my time, etc.

(side remark, I am in hindsight not very efficient here, but I think I've been better at it than doing nothing).

My current favorite agenda:

Somebody should be setting up an international moratorium on large training runs, improving human intelligence, and doing a mix of various agent foundation research + ambitious interpretability.

(side remark, most of my confidence here comes from deference/observing other people endorse the proposal)

This probably involves lots of techncial subprojects, e.g. model evals, ML experiments, compute governance, as well as non-technical subprojects.

I don't think I have enough information to sort out which small project needs more funding than others. Generally I think MIRI might know best where to direct money, so I plan to just donate to them.

(side remark to avoid wrong impressions: I haven't donated to MIRI yet because tax stuff and procrastination).

Overall I am not spending all my money on buying more governance & technical reseach, there's like some point where the best price for marginal utility lies in short-term interventions.

Voted no, but various preparations and adjustments can be made for "a world that is changing much more rapidly" - wouldn't advocate going "can't predict how it will go, so just carry on as if it won't make a difference", because that is definitely wrong. Figuratively, if you don't want to be caught flat-footed, stay on your toes. :)