For the purposes of this market, a "seed AI" is any AI that either creates discrete "child" AIs, or engages in extensive self-modification such that the post-modification AI is significantly more powerful/different. (e.g. the seed AI jumps to a larger server and runs gradient descent for 1 year).
If such a seed AI ever exists and it creates a successor / modifies itself, will the resultant AI be aligned (in the sense of having the same implicit or explicit utility function) with the seed AI?
In other words: will a seed AI that is facing its own version of the alignment problem solve it?
It is relatively unlikely that this market will ever be resolved, but please bet as honestly as you can anyway. Out of the goodness of your heart or whatever.
1,000Alignment seems really easy for an AGI. It can clone itself, check gradients, check the candidate on a wide variety of test inputs, property testing, search for compressed representations of novel architectures, handle partial correctness proofs by conditionally rejecting children on inputs they can't prove to be correct, etc.
With a little bit of selection pressure from even a dumb natural selection overseer so it learns to actually do all of these things, it seems plausible enough. (and with no selection pressure, early versions of the meta-algorithm would corrupt themselves into something not even functional on toy problems, and humans would remember to add it in)