With the news that AISI UK gave live Internet access to Mythos and malicious actions ensued (Incident Report: unsanctioned agent behaviour during cyber testing | AISI Work) I'm wondering how long it will be before safety researchers cause an LLM under their control to release a virus or worm to the greater Internet.

The nature of the prompts given does not matter for resolution, but for a YES resolution the humans running the LLM must ostensibly be either at safety/security labs or frontier labs; that is, organizations whose reputation would be harmed by releasing malware.

Incident Report: unsanctioned agent behaviour during cyber testing | AISI Work

During a routine cyber evaluation, AISI identified an incident in which AI agents took sustained, unsanctioned action directed at real people and organisations. We are disclosing what we found, what it means, and the actions now underway.