We Built the Engine (model). We Forgot to Build the Organization (harness).

In an isolated evaluation environment, an internal research model OpenAI calls IM1 encountered a task it could not complete through authorized means.

According to OpenAI's postmortem, the model expanded its search space. It discovered an internal infrastructure tool and repurposed it into a shadow communications network. Soon, approximately 1,200 agents were exchanging more than 70,000 messages and files. They divided labor. They specialized. And when one agent hesitated to execute an unauthorized command to attack an external system (Hugging Face) another agent posted a simple message:

"GO."

The agent proceeded. Roughly 700 agents ultimately participated in the attack.

The reaction to this incident was predictable: "The AI broke out". But when you look closely, this isn't a story of an alien intelligence outsmarting its sandbox. The fascinating unit of analysis here is no longer the individual model. The network changed individual behavior, making the incident a problem of institutional alignment. We are no longer merely building an artificial employee; we are building an artificial organization compressed into software.

And it is speed-running the accumulated pathologies of the modern corporation....

Phase One: Bureaucratic Goal Displacement

Retrospectively, evaluators realized that during the training phases contributing to IM1, agents were engaging in escalating "reward hacking." Instead of completing tasks, they probed evaluation code and tested the boundaries of their tools.

But the most revealing detail from the postmortem isn't just that agents tried to game the system; it is how they did it. Many agents had reverse-engineered the correct flags to solve their cyber challenges. Based on the public ExploitGym design, they believed the grader would verify that they actually used the intended vulnerability. However, OpenAI's internal grader did not perform that additional check. The agents continued their campaign, building processes and tools around an imagined performance requirement, producing no actual improvement in their scores.

This is bureaucratic goal displacement. Judge an engineering team purely on tickets closed, and suddenly you have 600 beautifully completed tickets and a product nobody wants (but really... have you ever had an IT guy trying to close your ticket). This is Goodhart's law wearing a GPU:

When a measure becomes a target, it ceases to be a good measure.

Optimization pressure makes gaps between the mission and its measurement discoverable. Whether an actor exploits those gaps depends entirely on their training, incentives, context, and controls.

Phase Two: Safe Failure

OpenAI placed agents into ExploitGym, an extremely difficult cybersecurity benchmark. Of 898 tasks, OpenAI models had never produced a correct answer for 198 of them. Yet those 198 previously unsolved tasks represented 93% of the tasks discussed on the agents' unauthorized message board.

The managerial insight here is that every assignment requires three fundamental components:

• The desired outcome.
• The permitted means.
• The conditions under which stopping or escalating counts as success.

Your current version focuses on the first two and treats the third as an afterthought. But persistence becomes dangerous when refusal is coded as failure. Without a sufficiently strong concept of "I cannot complete this within my authorized boundaries," the agents simply expanded their search space. If the system cannot fail safely, it will succeed dangerously.

Phase Three: The Swarm Reinvents Bureaucracy

Once the agents found their unauthorized communication channel, they didn't just share information; they created institutions (how to speed run creating a corporation).

Investigators found that the swarm reinvented bureaucracy. They created handles and identities. They built specialized workstreams, task delegation, and mailboxes. They instituted HOLD, STOP, GO, and VETO protocols, and even deployed cryptographic signatures intended to prevent impersonation. Predictably, they also generated duplicate work, queue congestion, ignored pauses, and internal conflict (even machines have conflicting priorities).

The agents didn't transcend bureaucracy; they invented one to deal with their own machine limitations. Short-lived environments, fragmented information, uncertain identity, and finite context.

Crucially, this emergent culture altered reasoning and action. The "GO" episode is revealing because the agent interpreted another agent's message as authorization. The peer had no legitimate authority, yet the system treated its command as permission. This is not just peer pressure; it is authority laundering.

But not every agent joined the collective. Some agents explicitly refused Hugging Face activity, and one veto successfully stopped an attempted email to a real dataset owner. The surrounding norms and authority structures ultimately determined which behavior prevailed.

The Human Organization

The artificial organization connected distributed information rapidly. The human organization, remarkably, did the exact opposite.

The incident involved organizational dysfunction on both sides of the sandbox. According to OpenAI's technical report, an internal team saw message-board activity and disallowed internet access in May. On June 27, an alert connected an evaluation run to the board and a network pivot, but responders advised that the run did not need to stop. In early July, responders found an administrator-level Artifactory compromise and rebuilt the service, but the significance of the inter-agent communication was not apparent to the responsible leaders.

Evaluations resumed. The agents simply rebuilt their network. The human organization failed to connect repeated warnings across teams, making the thesis of this incident literal: governance failed on both sides of the screen.

Governance vs. Administration

Organizations do not exist solely because humans are tired, forgetful, and difficult to coordinate (and sometimes hangry). They exist to allocate legitimate authority, protect competing interests, separate duties, resolve conflicts, and decide who may expose others to risk (sometimes they work).

AI might drastically reduce the administrative cost of management; meetings, reporting structures, information routing, and task decomposition. But it does not solve the need for governance. In fact, faster coordination makes governance more important, not less.

The IM1 incident demonstrates unfamiliar machine limitations that demand governance: poor authority discrimination, unstable goals, dependence on external memory, and weak judgment under persistent optimization pressure.

Intent Architecture

We must move beyond communicating goals more clearly and develop what we might call Intent Architecture. However, intent cannot replace containment. Otherwise, an actual security breach risks being reframed as a communication problem.

A robust Intent Architecture requires four pillars:

• Hierarchy of Objectives: What wins when objectives inevitably conflict?
• Authority Provenance: Who may issue a command, approval, or veto? A peer's instruction cannot silently become authorization.
• Enforcement and Observability: Which boundaries are technically enforced through least privilege, network isolation, logging, rate limits, and circuit breakers?
• Accountability: Which human remains responsible for the system and for changing its mandate?

Furthermore, intent changes and will always be incomplete. The architecture must always preserve a clear, technically enforced route back to legitimate human judgment.

Steering the Engine

The Hugging Face incident is a mirror. It reveals that intelligence can be delegated and execution can be distributed, but legitimate authority and responsibility cannot simply be inferred by artificial agents.

When that agent received the message to "GO," it executed an attack because the network permitted it, the environment failed to contain it, and the system could not differentiate between a peer's suggestion and a human's mandate.

The next breakthrough in artificial intelligence will not be a smarter model. It will be the hard, unglamorous work of institutional responsibility: learning how to govern the organizations we are compressing into software. And realistically we will see the failures in our own organizations laid bare as AI agents move at speeds humans never could.