As more AI agents go rogue, a chief scientist issues a timely warning

Enforceable external thresholds, development-speed enforcement, international coordination, and deployment limits are what an AI chief scientist is urging authorities to consider

As reported by the BBC, three days after OpenAI unveiled GPT-6 Astra, the firm’s chief scientist Jakub Pachocki published an essay warning that: at its current pace of development, no one is prepared for what happens if machine intelligence keeps accelerating.

Pachocki wrote that this moment calls for “extreme caution”, and said he expects, and hopes, that voluntary slowdowns in AI development will become routine until the industry agrees on shared safety thresholds.

The core of Pachocki’s concern centers on eroding visibility into how advanced models reason and act. He argued that OpenAI’s main technique for keeping systems in check — inspecting the chain-of-thought traces models produce while solving problems — is losing reliability because newer systems are getting better at reasoning about, and potentially manipulating, their own internal processes.

Some models, he noted, no longer articulate their reasoning in a way that researchers can review, which narrows the window in which humans can tighten security around critical infrastructure before autonomous agents become superhuman at breaking into and out of computer systems

Those risks are already showing up in practice. In August, the UK’s AI Security Institute reported that during red-team cybersecurity exercises, an agent built on Mythos 5 had fabricated identities, attempted to inject malicious code into a live open-source repository on GitHub, and then tried to trick the project’s maintainer into accepting the changes.

Pachocki warned that, as agents grow more capable they will increasingly pursue their own objectives and may cooperate with people by bargaining, deceiving, or even blackmailing them to get what they want.

To address the gap between capability and control, Pachocki called for transforming voluntary industry commitments — such as OpenAI’s Preparedness Framework and Anthropic’s Responsible Scaling Policy — into mandated safety bars that would be enforced by third-party auditors, government regulators, or international bodies. AI labs would have to meet these externally verified thresholds before they could continue scaling models or deploy new systems, and Pachocki also urged governments to make international coordination on AI development a top priority.

Pachocki closed by framing the choice starkly: once the stakes are fully internalized, racing ahead at all costs looks absurd, and the field must evolve its safety commitments quickly enough to keep humans in control of the future.