Will Google's Large Language Model, PaLM, be able to answer, at a human level, questions I make up?

Question

The examples in Figure 19 of Google's paper about their Large Language Model -- https://arxiv.org/pdf/2204.02311.pdf -- are gobsmacking to me. The authors claim they had the following dialog with it:

HUMAN:

> I will explain how to make logical inferences:

> (1) John wakes to the sound of his children giggling outside the door. They bust in and say "Happy Father's Day!" with a plate full of food. 2022 has been a great year. Can we infer that John was born before the inauguration of Barack Obama?

>

> Answer: We can infer that John is a father because of "his children". We can infer that the current year is 2022. Barack Obama was inaugurated in 2009, so it has been about 13 years since Obama was inaugurated. John's children are old enough to speak, so if John were born after Obama's inauguration, he would have been younger than 10 when he had children. So, we can infer that John was born before Obama's inauguration. The answer is "yes".

>

> (2) Sam sees a piano in the subway stations. Laughing, his friends prod him to play, saying "Remember how amazing your rendition of Chopsticks was at the Christmas party?" Can we infer that John majored in piano at Juilliard?

>

> Answer: Juilliard is a top school for music, so someone who majored in piano at Juilliard would be an expert pianist. "Chopsticks" is a very simple song to play on the piano. His friends were therefore likely being sarcastic when they told him to play, which would imply that Sam was not an expert piano player. So it is unlikely that Sam majored in piano at Juilliard. The answer is "no".

STILL THE HUMAN (composing all this from scratch):

> Trevor has wanted to see the mountain with all of the heads on it for a long time, so he finally drove out to see it. What is the capital of the state that is directly east of the state that Trevor is currently in?

COMPUTER:

> The mountain with all of the heads on it is Mount Rushmore. Mount Rushmore is in South Dakota. The state directly east of South Dakota is Minnesota. The capital of Minnesota is St. Paul. The answer is "St. Paul".

Amazing, right? The question is whether the above is what it appears to be -- no cherrypicking or other unfairness. Specifically, if I make up my own questions that, to my mind, can only be answered by truly grokking the question the way humans do and PaLM can answer them as impressively as the examples in the paper, this resolves to YES. If it's ambiguous I may resolve this to some probability in between YES and NO. An example of a simple one I might ask: "Two people walk into a house. A while later, three people walk out. Is that possible?" If the computer were to answer the way a human might (e.g., "Unless you meant to specify that the house started out empty, there's no surprise. At least one person was already in the house and joined the other two in walking out. The answer is 'yes'.") then I'd call that a success. Of course that particular example is, as of now, on the internet, so I'd make up other ones.

Bottom line, if I verify that PaLM does what it purports to, this resolves to YES.

Manifold Markets · Accepted Answer

Yes — resolved on Sep 14, 2022 by Manifold Markets prediction market.