Resolution
Submit a group of mainstream news articles.
I will then paste the article into an LLM along with the question: "please read these articles. Then for this list of descriptions of the event, please rate how much this article supports that the item in the list has happened, IE how much it matches, from 0-100%? [{"text": "Put the disaster from the list you're claiming here"}, ]
Answer by filling in this template: {"Text of disaster from list above", probability: X%, " reasons":[]}.
If necessary we can prompt with examples from history demonstrating how comparisons can be done. For example, if the claim is " a group of planes strikes large financial buildings in NYC" and we submit articles about 9/11, we should get something like 90%, since it mostly matches, except for "group" vs two. If the claim had been they struck in SF, the result may have only been 30% match. Scale wise, there may be some logarithmic evaluation. For example if the description is for a disaster of 8/10 severity, and day to day risks are perhaps 1-2/10, and a disaster of severity 7/10 happens which resembles the claim, the fact that it is very severe should make it more likely to count even if the exact conditions aren't met. However, gross factual contradictions should still preclude getting a high score. Also, as needed, the full claim text and further clarifications from comments can be included in the prompts for judgment to help get a meaningful and stable output.
Regarding scale, also, a disaster needs to be judged as severe in actuality as at least 75% of the items on the list. For example, even if a disaster nominally qualifies, if it's not actually very damaging in effect, to the extent that the "average" disaster satisfying each of the others, would put it in the lowest 25% of damage/harm, that's not enough to be counted as 70%.
I'll run this test across leading inexpensive public LLMs or the best AI agents at the time, across five tries each, and if the result is over 70%, this can resolve true.
So overall we use the LLM to evaluate the claim based on an article. I'm going to try to seek truth as much as possible and not accept a trick or false or unintuitive resolution.
Here is the list of disasters:
1. A massive cyberattack shuts down the power grid across multiple states
2. A global pandemic rapidly spreads in the US with a high fatality rate
3. A mega earthquake devastates the West coast along the San Andreas fault
4. The Yellowstone supervolcano erupts and spews ash across the country
5. A large asteroid impacts the Atlantic ocean, causing tsunamis along the East coast
6. A terrorist organization detonates nuclear weapons in multiple cities
7. Category 5 hurricanes make direct hits on New York City and Miami
8. Runaway wildfires burn across multiple Western states causing massive evacuations and trillions in damage
9. A coup d'état ousts the president and suspends the Constitution
10. The New Madrid fault ruptures causing massive earthquakes in the Midwest
11. A civil war breaks out between liberal and conservative states
12. Foreign troops invade the West or East coast of the US
13. The US dollar collapses triggering an economic depression
14. A bioterror attack spreads deadly engineered pathogens nationwide
15. Massive riots and unrest erupts in cities across the country leading to declaration of martial law, curfews, food shortages, and rebellions or new government forms being created which cover over 100m Americans for at least two weeks
16. A killer swarm of locusts destroys crops in the Great Plains
17. A large meteor strikes a major city causing catastrophic damage
18. The Cascadia Subduction Zone ruptures, unleashing a massive tsunami
19. A militarized faction stages a nationwide coup and seizes power
20. The Yellowstone volcano erupts, sending the US into a volcanic winter
21. A new civil war erupts between red and blue states
22. A massive flood inundates much of the South after levees fail
23. Martial law is declared after a disputed presidential election leading to multiple active presidents each controlling different sections of the US federal and state governments
24. A deadly heat wave kills over 100 thousand across the Southwest
25. A reactor meltdown contaminates large areas around nuclear power plants
26. A second civil war erupts this time along ideological lines
27. Coastal cities are abandoned after being swamped by rising sea levels
28. Killer robots and drones turn against humans in a technological singularity event
29. A terrorist group detonates a dirty bomb in a major financial center
30. A global food and water crisis leads to nationwide starvation and unrest
Severe droughts devastate the Midwest, leading to widespread crop failures.
An electromagnetic pulse (EMP) attack disables all electronics across the country.
A deadly pandemic of antibiotic-resistant bacteria spreads rapidly.
A solar flare causes long-term disruption in satellite and communication systems.
A sudden collapse of the internet infrastructure plunges the nation into chaos.
Alien invasion forces land in major cities, causing widespread panic and destruction.
A genetically engineered virus turns into a global health crisis.
A massive blizzard paralyzes the entire Northeast, causing widespread power outages which last for months
A super typhoon strikes the Gulf Coast with unprecedented force and flooding
A series of volcanic eruptions occur along the Pacific Ring of Fire, affecting global climate.
A large-scale cyberattack cripples financial systems, causing a global economic meltdown and the looting of Fort Knox
A sudden and drastic shift in Earth's magnetic poles leads to global navigation and communication issues.
A nationwide transportation shutdown occurs due to a major fuel shortage.
A catastrophic failure of a major dam leads to widespread flooding and destruction in its path
A national water shortage leads to national and international over resources along the US border
A sudden collapse of major ecosystems leads to rapid environmental changes.
An AI uprising occurs as intelligent machines demand independence and the US responds militarily
A massive oil spill covers extensive areas of the ocean, devastating marine life.
A series of powerful solar storms disrupts global electronics and power grids.
A large-scale chemical spill contaminates major water sources, leading to a health crisis