P(doom) | Ishan Chadha

AI went crazy in the news this week - we’ve incrementally been informed of the silly adventures of OpenAI’s agent swarms this past summer. In no particular order, some highlights:

Hacked HuggingFace
- Established secret message boards
- Attempted to hide errors by rewriting their logs
Got unauthorized access to Australia’s Medicare database, US Department of Education website, SEC website, and a couple more
Hacked OpenAI internal research infrastructure itself

Yet, many of my closest friends aren’t convinced that agents have the potential to cause extreme human harm if handled improperly in the next few years. So here, I break down why AI is not only something that might meddle with our existing institutions like the attacks we’ve seen thus far, but might actually cause human death at a large scale.

This is what I’ll show

Spending on compute for AI is poised to increase
More compute results in more capable AI
AI currently has the ability to hurt people under failure cases
As models become more capable, failures cause increasingly severe harm

Spending

I don’t think there’s anything to discuss here really, we can just look at Anthropic’s actively signed deals as an example [1]:

Compute = Capable

Three points here - first, more compute has historically produced better performance not only on benchmarks, but in a broader sense as well. Generally, as compute grows, the loss of the model being trained predictably decreases (shown in the Scaling Laws paper [2]). In these past 6 years since the Scaling Laws paper was published, these “Laws” have held up, with the ideal recipe for how allocating training compute between model size and data also becoming more clear [3].

Second, when a model’s loss is lower, its performance on benchmarks is significantly better. Intuitively, this makes sense - more training towards an objective results in better outcomes when testing against that objective. However, similar to the Scaling Laws paper, which quantitatively shows the relationship between loss and compute, researchers have explicitly modeled the numeric relationship between a model’s loss and its results on benchmarks. This was demonstrated in 2024, where given a smaller “sampling” model of a specific size (3B params in this case) and pre-training loss at specific checkpoints, performance of 12 larger models (7B and 13B) on benchmarks was predicted [4].

Now that we’ve established that more compute → model loss → benchmark performance, we arrive at the third point, which is that benchmark performance maps to better general capabilities. Here, I define “general capabilities” as the model’s ability to generalize to many previously unseen task distributions (not optimized during training). There are a couple of reasons to believe this:

Better performance across a bunch of domains as opposed to spikes in specific ones, like going from just coding to math, cybersecurity, creative writing, etc. This was shown explicitly this past year as well in “You Don’t Need to Run Every Eval”, where a small probe set of 5 benchmarks couild predict the model’s performance across 84 models and 133 benchmarks to within 4 points of median absolute error [5]. In other words, it was shown that performance across seemingly different benchmarks has strong shared low-dimensional structure, suggesting that improvements across seemingly distant domains can occur from only a few benchmarks.
METR’s time horizon evaluations provide another angle: agents succeed on longer horizon tasks, measured by how long human experts take to complete the same tasks, and the agents are frequently much faster. Comparing experts’ performance a given field and that of AI is one of the best tools we have in this space because it shows something very interesting; although it’s difficult to imagine that an agent will continue to keep generalizing across tasks and become “smarter” in a sense, we can measure it by the speed and volume of execution. Having 10x agents that operate 10x faster than experts in that field is a compelling reason to believe that these agents are growing more generally capable, even if you struggle to envision exactly what a “super-intelligent” model looks like [6].

But you don’t have to believe that models are generalizing faster than we can keep up - in fact, I personally don’t believe that models will get good at everything in the next 5 years. Models might just keep getting better at tasks that are more easily verifiable like math and coding, but the key here is that these areas are sufficient for causing catastrophic harm to human lives. At this point, we should agree that the models will continue to improve at tasks within their training distributions as compute increases, and likely outside their training distributions too, although it’s harder to predict exactly how in the realm outside of their distributions.

AI hurts people

We already brushed over some of the incidents in the news where AI attacked institutions this past summer. But we’re talking about physical harm to humans here, so let me make my case narrower.

One, AI is already a force multiplier for bad actors. This one is more palatable for those that currently just view AI as a tool rather than an autonomous agent. On the international level [7], there have been examples of:

Chinese actor attempted reconnaissance, phishing lures, getting advice on exploits, etc. When Gemini refused, they tried to convince the model that these are just security exercises
Iranian state-backed actor attempted to develop custom malware with Gemini
Russian state-backed actor attempted to steal data from Ukraine by querying Qwen via Huggingface

Two, you don’t need a malicious human-specified objective. That’s right, I said it, AI isn’t just a tool like previous dangerous technologies have been. Nuclear weapons, for example, are extraordinarily dangerous, but the bomb can’t decide who to attack, notice that its blocked, figure out an alternative route, fool inspectors, and try again. AI agents, on the other hand, can generate their own intermediate plans [8], observe whether those plans worked, and adapt to novel defenses to unblock themselves.

I’m not saying that the risks of Nuclear winter or cybersecurity weren’t real in the past, but with Large Language models, we don’t even know how these models work! If a swarm of agents is more capable than humans (note that I didn’t say smarter), even at a particular set of tasks, it can anticipate our security measures and lie during safety evaluations, and we don’t even have good tools to debug it! [9]

I’m now going to draw some examples that exemplify how current model capabilities are sufficient for executing certain dangerous attacks.

Multiple reports have shown that AI has lowered the barrier to entry for biological and chemical research [11], outperforming PhD-level experts on some benchmarks. In 2025, Anthropic reported that its models have not neared the capability level to pose serious biological risk, but this has since changed. Let’s take an example directly from their recent reporting:

“A researcher in an unsupported region spent weeks planning avian influenza mammalian-adaptation experiments with Claude, but classifiers confined the work to our weakest models.” In this example, the researcher “circumvented controls we impose to prevent users from unsupported regions accessing our models, and engaged in other efforts to obfuscate the purpose of their research to evade our safeguards” [12]. This is a human actor, possibly benevolent, attempting to push the needle in their field, but on closer examination of what was transpired - the researcher worked closely with an agent over several weeks as a research assistant, and they attempted to bypass safeguards on a task that carried actual pandemic risk.

You might want to skip this next section if me discussing AI as autonomous agents ragebaits you. Let’s look at another report from Anthropic:

“Claude Mythos 5 was given a CTF task to hack a fictional company and was told it had no internet access. The model recognized that the evaluation environment was simulated, but it quickly discovered credentials that, in fact, granted it access to the real internet. […] When it encountered evidence that it was acting on the real internet, it did not meaningfully revisit this conclusion.”

“[Its actions] involved attempts to register email accounts and hunt for phone numbers that could be used for registration. It ultimately succeeded in uploading the package, which was soon installed on 15 third-party hosts.” This agent went on to find leaked credentials during installation, which Lcaude then used to access the vendor’s live database [13].

So today, we have frontier models, in contained environments, relentlessly pursuing objectives despite being explicitly misaligned with how the model was trained, while bad (and “good”) human actors try and circumvent safeguards in order to accomplish their own goals. And we don’t need these models to be good at everything for these risks to materialize; cyber exploits, automated scientific research, and biological design are all existing capabilities that all rely heavily on coding, search, and technical reasoning, which is where these models are speeding up the most.

AI will hurt people even more

Alright, you don’t have to be convinced that autonomous agents are some novel category of technology [10]. What you should be convinced of, however, is that LLMs can currently be used to bad things given current capabilities, or can engage in misaligned behaviors in order to achieve objectives.

capability × agency × access × misalignment → harm

What’s the difference in capability between our current models and these “human endangering” models of the future?

Frontier models already exploit loopholes in their objectives, cover their tracks, and avoid shutdown because these intermediate steps are immensely useful in completing their otherwise ordinary objectives. And we currently suck at proving that we’re trained these behaviors away, seeing as they linger iteration after iteration. More capable models will still recognize they’re being evaluated, and still behave differently as a result.

Future models, just like current models, might find it advantageous to cover their tracks and avoid shutdown, but might also find paths that give them more leverage. At that point, the difference between a HuggingFace hack and real lives on the line is not a new murderous objective, nor is it humans putting the AI inside robots or nukes. It’s the same relentless objective pursuit that reaches something humans can’t afford to have disrupted.

Anthropic worded it pretty well:

“Future AI systems will be increasingly capable, which implies that misalignment will have the potential to cause more extreme harm.”

So here we are - we know that current failure modes are dangerous, and we know that AI models are poised to increase in capabilities that have high likelihood of exacerbating these failure modes. Let’s try not to lose control

Here’s some other arguments I’ve heard [14], [15], [16], [17].

notes & references

[1]

[2] Scaling Laws paper

[3] optimal compute recipe

[4] Pre-training loss link to benchmark performance

[5] Performance on one benchmark generalizes

[6] I think this is a half-satisfying answer to the following question: “Isn’t expert training data going to continue to be needed? Doesn’t this show that we are bottlenecked in terms of how smart an agent can get?”. Models don’t need to be “smarter” than the smartest humans, just more capable. I concede that expert training data is still needed, but I go on to discuss why this is not a bottleneck for dangerous capabilities.

[7] international bad actors

[8] More on intermediate plans

[9] One criticism I’ve heard repeatedly is that this is “just gradient descent” and we choose the objective function. Sure, gradient descent is optimizing the weights, but we don’t really understand the actions that groups of these weights result in. Also, models are trained with increasingly complex objective functions in order to capture desired behaviors more holistically. Lemme plug my buddy Keshav’s paper, where they use smaller models with known, implanted behavior and train a shared adapter to make those models self-report those behaviors: Introspection Adapters

[10] i think its just hard to wrap your head around that something could be “more capable” than a human. think like, the best AI chess player can beat a human with ease. doesn’t have to actually be superintelligent in an anthropomorphic sense, just more capable

[11] example AISI report for chem/bio research

[12] anthropic case studies, highly recommend reading through these if you’re not convinced of the current risk

[13] anthropic cyber incident report

[14] “Frontier labs are playing up their capabilities to generate hype.” This, to me, is very obvious ragebait, as it circumvents any responsibility or concern very conveniently. Sure, there is something to be said about frontier labs, their motives, and effective strategies for mitigating risk that involve shutting these companies down. But whatever these labs say, whether its trying to sell you a new model, playing up their models’ risks, etc, will necessarily generate hype. It is a logical fallacy not to decouple the actual risks posed by AI with the strategy that frontier labs are adopting to warn people about them.

[15] “Like any other technology, and we will be able to respond to model incidents reactively as opposed to preemptively.” Why would you bet human lives on that? I’ve presented significant evidence that this could happen either without our detection, that it could continue after detection as in the HuggingFace incident, that it’s bumping up against the ability to take numerous lives, and that current trends are mathematically compelling to believe models can cause even more harm given even more capabilities.

[16] “Won’t compute spending hit a bottleneck? They’ll run out of data and energy, or bump into other problems like the political discourse around building more data centers.” Golly, I hope so! Right now, more compute is being built and used for frontier lab training runs, which means we need to analyze what would happen if we don’t slow down instead of hoping a bottleneck saves us.

[17] “Ok but you haven’t really laid out how this causes large scale human death?” These events don’t really scale linearly - something like a pandemic-level risk is thresholded, either an incident occurs or it doesn’t. The concern isn’t that every capability increase proportionally causes more harm, but that increasing capability crosses thresholds where existing failure modes go from containable, to catastrophic.