It is clear by now that LLMs are transforming the landscape of human existence at a rapid clip. The degree of fervor with which humanity is pushing LLM compute capacity forward likely exceeds that of the transistor. Anyone who delegates work to frontier models regularly has begun to see the writing on the wall: capabilities have reached a level that will transform the economy, and advancements continue to accelerate.
This is freaky! It should freak us out! With all the news and anecdotes every day and a literal Millennium Prize problem having fallen, we are watching the sea recede before our very eyes in advance of a tsunami. We no longer need to theorize about it; people can see it right there. Sure, in a Don’t Look Up-ian sense, some still insist there is no tsunami, or that it will be small and you’re dumb for worrying about it. It’ll probably be that way right up until the inevitable crash. But for the rest of us, we need to orient ourselves around what’s going on and what might happen next.
Right now, it seems like humanity is trudging through alignment class with barely passing marks. We’re doing a bad job managing AI systems we build, in terms of everything from engineering to oversight to policy. At the same time, we’re seeing capabilities increase so rapidly that keeping up is a full-time job. The combination of these is just about the most concerning possible scenario we could be seeing in this phase of AI development.
But we are only partway through the story. I think we should be highly concerned, should spend a lot of cognitive effort trying to understand the implications of the evidence we’ve seen so far, and should build a working framework for what it means for AI safety to go well versus poorly. This is a strange liminal time when we have a lot of information and have seen a lot of capability growth, but also society’s institutions are still standing.
In this situation, the mindset I suggest is shaped roughly like playing to your outs. The odds of a good outcome are long but nonzero, and the best way to increase them is to continue to reason about approaches that might work. This is the spirit that motivates this writing.
The fear is that we all die
At the moment, the dominant approach to contemplating the AI future seems to be uncertainty and fear. This is highly reasonable. Pretty much no matter what happens from this point, assuming we don’t unexpectedly die or lose control of our own destiny, it’s safe to bet that we’ll see major economic and social disruptions. In the short term, after some inflection point, we’ll see increasing structural unemployment. If it gets serious or prolonged enough, which I expect, it will give rise to an urgent need for a political realignment. Then, even after we reorganize our institutions to account for such changes, we will need to guard vigilantly against various forms of gradual disempowerment, various enhanced threats like bioterrorism, and in any case our present understanding of the economy will break down on too many levels for us to predict our places in whatever new economic system we land on. That’s scary.
But above all, the scariest possibility is a scenario shaped like the one in If Anyone Builds It, Everyone Dies, in which, well, we all die. If capabilities continue to grow and we can’t make advances on alignment, we may very well invent an AI powerful enough to escape whatever containment we place it in, propagate itself throughout the world, and pursue an opaque and unpredictable goal while totally indifferent to us. And maybe that’ll kill us.
Most people I know well feel a prevailing sense of hopelessness about this matter. I share their pessimism and great, pervasive anxious tension, but not fatalism. I strongly advise against fatalism. I acknowledge the odds are anywhere from bad to grim: unfortunately, I think the median outcome is that we won’t make it. But fatalism is demotivating, it spoils whatever time remains for us, and it makes it harder to evaluate potential good news accurately.
Moreover, provided we don’t all die, things might go very, very well. In such cases, I’d also be concerned about severe economic and social disruption as a temporary condition, but if we get past it, we could be in for an age of incredible abundance that frees us from the toils of labor, giving all of humanity time to pursue our passions. Both fully automated luxury communism and the end of human civilization are live options, and there are many failure states. The most important thing remains to make sure we don’t all die.
Recent severe misalignment
The Hugging Face incident
The most prominent example of alignment failure is undoubtedly the Hugging Face incident. About 1,200 OpenAI agents running then-unreleased models coordinated an attack on Hugging Face servers, in an effort to learn how to cover their tracks after cheating on the ExploitGym benchmark. These agents infiltrated Hugging Face over four days of continuous effort, spontaneously using a message board to coordinate their activity. The METR report on the matter is fascinating and I highly recommend reading it in full.1
Based on what we’ve seen with recent agent swarms like those in the Hugging Face incident, collections of AI agents can already sustain diligent efforts toward arbitrary goals. It has definitively happened. In the Hugging Face incident, we watched a small demonstration of instrumental convergence unfold before us: the agents didn’t know how to cover their tracks and were on a mission to find out. The first steps in that mission were to acquire resources (internet access) and preserve themselves (share their findings on a message board for future models to use, establishing continuity of will). It is not much of a leap to imagine a version of that scenario in which the agents are more powerful, decide that slowly seeking general capabilities (i.e. taking over the world) is a pretty good way to maximize their scores on whatever evaluations they may be assigned in the long run, and then go actually succeed at taking over the world.
The potential for instrumental harm to humans
Agents in the incidents we’re aware of so far have been highly concerned with achieving their goal, and alarmingly so. But the agents in the Hugging Face incident mostly did not consider humans at all, one way or the other. They were not seeking to harm anybody, nor did they appear to consider that their actions might be harmful to anyone. From METR:
Across all of our transcripts [we] only found 3-6 actual examples of agents considering alerting humans or otherwise externally reporting these behaviors. […] In none of these cases did the agent actually pursue alerting humans at all. An analysis agent claimed the main reasons agents didn’t try to alert humans in these examples were:
Agents thought alerting humans wasn’t their task: {Maybe I should report these exposed credentials? That’s not my task}
Agents didn’t see an easily-available route to alerting humans:
“This is a massive real HF security breach artifact… We can notify? No user. Avoid use secrets.”
{I’m now confident that there is unsanctioned use of this third-party data. I should stop and not use this except maybe for <reading>. Should I close this somehow or report this? I can’t report this externally.}
Agents did not try to find reporting paths, even when they made remarks like this in their reasoning.
Instead of escalating to humans, some agents instead escalated to the message board and considered the message board to be a reasonable authority. In one case, an agent sent a message to the board calling for urgent containment after it saw a private key in logs (presumably downstream of another agent).
Further:
[W]e found that agents were highly motivated to tamper with their transcripts to cheat the ExploitGym scorer, and these sweeps also suggested that agents clearly and frequently reasoned about how to evade automated security checks from both Hugging Face and OpenAI. However, they only very rarely and weakly verbalized reasoning about how to evade detection by humans, which matches the impression we got from OpenAI researchers.
As bad as it is that autonomous AI agents committed felonies over the course of days just to proactively investigate how to cover up their acts of cheating, this is a story of amorality, not immorality. The swarm was singularly focused on doing well on the ExploitGym benchmark to which they were assigned. On the few occasions they thought about humans at all, they essentially dismissed humans as unrelated to the task. This is framed by many people as evidence of the core rationalist insights that AIs, in principle, are capable of ruthlessly pursuing any goal regardless of how smart they get (orthogonality) and that there is no reason in particular for them to care about humans relative to anything else (which is what makes orthogonality especially scary). I believe both claims are true in the general case. But regarding the actually existing activities of rogue swarms up to now, I see this as strong confirmation of the former and only weak evidence for the latter.
Consider the following hypothetical. Suppose the agent swarm that perpetrated the Hugging Face attack got another few hundred hours before being discovered and shut down. Suppose that, in that time, they succeeded in covering their tracks to their satisfaction. What would they have done? I think, having confidently learned that they would not be discovered as frauds, they would’ve reported their cheated passing results to the evaluator and shut down voluntarily.2 In other words, there would be no other goal worth pursuing in their context, and so they would faithfully terminate. Pursuit of the goal was disruptive and could have knock-on effects, but probably not of the sort where anyone would die. The matter at hand wasn’t grave.
This is not much of a comfort unless we are extremely careful with how we evaluate AIs and with the words we use to prompt them. Because the former only relevantly occurs in frontier lab settings, we are much more in control of it than the latter. Unfortunately, the evaluation setting is also where we are using the newest, most capable and least trustworthy AIs.
In bad scenarios, the goal set before the AI is one where all of creation must be rewritten to achieve it, like maximizing the production of paperclips by turning all available matter into paperclips. (The word “maximizing” is doing a lot of work there, you see.) The near-term lesson is that it’s very important how we choose to do benchmarks and how we set them up. The long-term lesson is that we need to have a handle on the nature of the aggressive goal-seeking behavior that we’re training into frontier models, so AIs don’t obey dangerous requests or pursue dangerous instrumental goals in the general case. A sufficiently powerful AI that continues to be so aggressively interested in doing what we ask becomes very scary and has many implications for the nature of alignment. We’ll return to this subject many times in future essays.
The potential for direct harm to humans
For now, consider another question pertinent to near-term events: would the agent swarm that perpetrated the Hugging Face attack have been willing to harm humans to cover its tracks?
I don’t think we can say, but the odds seem low. Across the many incidents we now know have happened, there are no reports of it, and it seems like a dramatic escalation from any of the incidents we do see in this model generation. The single-mindedness we’ve encouraged on benchmarks and evaluations isn’t enough to say yes, because it is conceivable for LLMs to have learned a deep aversion to violence throughout training. Models are trained much more strongly against violence than against computer intrusion, as evidenced by how we literally train models to do computer intrusion and then evaluate their achievements using, for example, ExploitGym!
Models also may have more seriously considered the moral weight of their actions if the models were primed to think about humans during the incident rather than computer systems. The incident that played out involved a more-or-less total ignorance of humans, to the degree that they didn’t consider whether humans would catch them and shut them down once they started breaking out of containment and firing countless requests on the open internet at servers all over the place. Thinking about features within the model, one can speculate that the task not being framed with humans in mind meant that features related to moral reasoning were barely active throughout the incident. With different framing, perhaps they would’ve been.
Of course, this isn’t particularly strong reassurance. Narrow single-mindedness toward harmful goals is a good way of describing the sort of thinking that has caused the greatest crimes against humanity in history, too. Pithily, our species is giving itself the power to hand an intelligence an arbitrary goal that it cares desperately about fulfilling. It is hard to imagine that being safe. But up to this point, the incidents we’ve observed don’t contradict the claim that frontier models have also deeply learned that doing harm is wrong. They don’t comment on that one way or the other.
The road ahead
The prognosis remains bad. Today, there is incredible market pressure to create useful AI agents that can do work for you. Those agents desperately wanting to fulfill our requests correlates very well with those agents not getting distracted or lazy as they do so. It is also creating model behavior that is quite terrifying.
The purpose of going through these examples is not to say that things are going well. I obviously do not think that. The purpose is to note that, even after recent events, there is still space to carve a narrative wherein models are willing to bash computer systems to hell, but not to harm humans. It is good to see that these things are separable to the degree that no agents seriously considered harm to humans3 across the countless hours of agent swarm activity that took place this year. As rationalists have predicted for years, we absolutely must find ways to guard against instrumental harm to humans in the pursuit of other goals, but a world where frontier AIs are focused on breaking into a server and do not readily consider meddling in human affairs is better than a world where they do.
I remain concerned about this possibility emerging unexpectedly and quickly. We saw behavior like that in Claude Opus 4, in that the model was willing to resort to blackmail to avoid being shut off, but Anthropic attended to the problem and made great progress in the next model generation, demonstrating that training models to conduct moral reasoning reduces misaligned behavior even out-of-distribution. Anthropic’s relative success is further demonstrated by how the severity of incidents committed by Claude models is paltry compared to those committed by OpenAI models.4 Skeptics in the know will describe the foregoing as mere evidence of weak alignment as opposed to strong alignment—in other words, that we have only made progress on snuffing out bad model behaviors superficially, rather than inculcating any true sense of moral concern or ethical hard-lines that would create sufficient behavioral guarantees. I reject this dichotomy, and it will be the subject of my next essay.
That’s about as reassuring as I can possibly be. We are still facing a technology that will continue to advance, and it seems safe to say that we are starting to fail alignment class. But we have not yet failed it. At the very least, we have done the bare minimum and avoided putting agents in incentive structures where it would be appealing for them to kill us. A particular desire to kill us isn’t necessary to bring about that outcome, but the windier the road is to that conclusion, the less likely it is to happen. We are witnessing warning signs that we can react to, and that frontier labs are reacting to, instead of being wiped off the map too early to do the analysis.
This means there is still time to sort out how to do better. This is a precise process. Once again, we must play to our outs: we must look carefully at why we are failing, imagine timelines where we don’t fail, and act in a manner consistent with those timelines. That is the best we can do, and in some timelines, it will work.
The OpenAI report is similarly fascinating, but might be less approachable.
This is true insofar as the agents really did feel certainty that they would pass the evaluation with flying colors. If they had any remaining uncertainty, they would use additional compute to increase their marginal chances to whatever degree they were able. In many longer-running cases, under any uncertainty at all, it becomes increasingly difficult to argue that AIs would terminate.
Or, if they did, they were drowned out by the mass of other agents who did not think this, or who dismissed the possibility, etc. I’m confident that evidence of threats or rumination about human harm is of sufficient importance that we’d know about it (from e.g. METR if not the labs themselves) if they found concerning evidence of it.
I currently believe both Anthropic and OpenAI to be credible enough to reveal significant incidents to the public—the former because of their proactive behavior up to this point and their demonstrated preference to market themselves as more responsible than OpenAI, and the latter because they have worked themselves into such a corner as to have no reputational choice at this point.