—- AI Capabilities—

LLMs have a few different shortcomings :

Hallucinations
Context Windows
Mathematical and Logical Problems :

Will a prompt that enables GPT-4 to solve easy Sudoku puzzles be found? (2023)50%

#Technical AI Timelines

Will Mira's main GPT-4 Sudoku market ever again drop below 50% and stay there for at least 24 hours?NO

#GPT-4 Sudoku Challenge (2023)

#New Year's Resolutions 2024

#New Year's Resolutions 2024

[Math test 1] What will GPT-4 Vision score on the Mathematics section of JEE Advanced paper? (out of 60) ($200M sub)

15%

<0 (i.e. negative score)

29%

0-10

16%

10-20

12%

20-30

See 4 more answers

[Math test 2] What will GPT-4 Vision score on the Mathematics section of JEE Advanced paper? (out of 60) ($200M sub)

27%

<0 (i.e. negative score)

24%

0-10

21%

10-20

12%

20-30

See 4 more answers

[Physics test 2] What will GPT-4 Vision score on the Physics section of JEE Advanced paper? (out of 60) ($150M sub)

34%

<0 (i.e. negative score)

17%

0-10

18%

10-20

8%

20-30

See 4 more answers

#Competition Math

[Chemistry test 1] What will GPT-4 Vision score on the Chemistry section of JEE Advanced paper? (out of 60) ($150M sub)

9%

<0 (i.e. negative score)

17%

0-10

27%

10-20

11%

20-30

See 4 more answers

#Competition Math

[Chemistry test 2] What will GPT-4 Vision score on the Chemistry section of JEE Advanced paper? (out of 60) ($150M sub)

7%

<0 (i.e. negative score)

14%

0-10

36%

10-20

13%

20-30

See 4 more answers

Will GPT-4 fine-tuning be available by EOY?NO

Will a large GPT-4 equivalent competitor model be revealed by the end of 2023?NO

#Technical AI Timelines

Is GPT-4 best? (2023)YES

#New Year's Resolutions 2024

Will hallucinations (made up facts) created by LLMs go below 1% on specific corpora before 2025?41% chance

Will LLM hallucinations be down to human-expert rate within months?NO

Will scaling current methods be enough to eliminate LLM hallucination?26% chance

#LLM & AI Capabilities

Will we get hallucination rates down to human-expert levels by May 2024?NO

Will A.I., "Hallucinate Significantly Less," by the End of 2023?NO

Will there be a version of GPT4 with a context window of 1 million tokens this year?NO

Will any LLM have roughly GPT-3-level losses with a context window of at least 50,000 tokens before April of 2024?80% chance

#LLM & AI Capabilities

Will Safari include a chatbot at end of 2024?35% chance

Will any LLM be able to multiply together arbitrary decimal numbers by the end of 2027?75% chance

Will Firefox include a chatbot at end of 2023?NO

#New Year's Resolutions 2024