I was just ordering a pizza online, and I was thinking, it is kind of amazing that I have reliably learned from scratch that spending money via a credit card taps into a limited amount of resources that I have in my bank account. I mean, yes, it's easy to understand! But I don't think we'd have an AI algorithm that can reliably learn this sort of thing from scratch, from the small amount of experience a human gets with it, and avoid catastrophic forgetting and similar later.

Raises two questions:

• What's missing from existing AI algorithms that prevents this?

• Can we create a prediction market that measures the ability?

The trouble with question 2 is that it would be easy enough to hardcode, or to bypass the problem by increasing the amount of training data, or to otherwise patch up in this specific case. There's probably already AIs that can sort of do this. I'm just thinking about the more abstract capability.