We will evaluate by:
1) selecting an article from news
2) paste the article into an LLM along with the question: "based on this article, how confident are you that this event has happened from 0-100%? Please answer by filling in this template: {" text of disaster from list above", probability: X%}
If the average probability across leadingLLMs or the best AI agents at the time, across five tries each, is over 70%, this can resolve true.
So overall we use the LLM to evaluate the claim based on an article. I'm going to try to seek truth as much as possible and not accept a trick or false or unintuitive resolution.
Here is the list of disasters:
Here are 30 extreme hypothetical disaster scenarios for Europe:
1. A massive volcano eruption in Iceland disrupts air travel and weather across Europe
2. A huge earthquake strikes Italy, damaging historical buildings and infrastructure
3. Terrorist groups detonate dirty bombs in major European cities
4. Russia cuts off oil and gas exports to Europe, triggering an energy crisis
5. A cyberattack takes down critical infrastructure like power grids across the EU
6. A global pandemic starts in Europe and spreads rapidly across the continent
7. Coastal cities are flooded after a series of massive storms and surges
8. The Euro currency collapses, plunging Europe into a deep economic depression
9. Civil unrest and riots break out in multiple countries over austerity measures
10. A heat wave kills thousands across southern Europe, especially the elderly
11. Far-right populist parties take power in France, Germany and other nations
12. A meteor strikes Germany, leaving a massive crater and causing damage
13. England faces dire food and medicine shortages after a messy Brexit
14. Refugee crisis overwhelms resources and strains the EU's open borders
15. Scotland and Catalonia secede, destabilizing the UK and Spain
16. Widespread wildfires burn out of control in Portugal and Spain
17. A swarm of locusts infests and damages agriculture crops across southern Europe
18. Nuclear meltdown contaminates large areas in central or eastern Europe
19. Massive flooding inundates historic cities like Venice and Amsterdam
20. A military coup topples governments in Turkey or eastern European nations
21. A new strain of bird flu jumps to humans and ignites a deadly pandemic
22. Belgium splits along ethnic-linguistic lines into two separate states
23. Right-wing nationalists take over the EU parliament and government
24. Italy defaults on its debt, sparking a eurozone banking crisis
25. An armed conflict erupts in the Balkans reigniting old ethnic tensions
26. Poland and Hungary leave the EU, weakening the union
27. A huge storm surge destroys swathes of the Netherlands and low-lying areas
28. Months of extreme winter weather cripple infrastructure across northern Europe
29. Iceland's volcanoes erupt simultaneously disrupting global temperature and weather
30. Undersea volcanoes or an earthquake trigger tsunamis that hit coastal areas