The AI safety debate: Why ‘pacing the frontier’ comes with pitfalls - The Indian Express

For an industry built around a race to make its models more powerful, calls from some of its biggest executives to deliberately slow down have opened up an unusual divide in Silicon Valley and Washington DC. Particularly over how real the dangers from advanced artificial intelligence (AI) are, who should regulate them, and whether the US can afford to apply the brakes while competing with China.

Anthropic chief executive Dario Amodei’s proposal to “pace the frontier” has found support in places that would have appeared unlikely even a few months ago. OpenAI chief Sam Altman, one of Amodei’s fiercest commercial rivals, has said his company also believes frontier development may need to slow. Microsoft chief Satya Nadella has backed “deliberate pacing” and independent evaluators. 

On the other side is US President Donald Trump, whose administration has increasingly treated AI as a technological contest in which American dominance over China takes precedence over new regulatory restrictions. Trump has called fears that AI could take over the world a “hoax”.

Meanwhile, some prominent technology executives and investors argue that warnings from the largest AI companies could end up giving those same firms a regulatory moat against smaller competitors.

The argument is therefore no longer simply between AI “doomers” and “accelerationists”. It now encompasses several overlapping questions, including whether AI systems have become dangerous enough to justify slowing capability gains; whether voluntary promises by laboratories are sufficient; whether governments should impose restrictions; and if any US slowdown is credible if China does not accept comparable constraints.

Amodei’s proposal starts from two developments. The first is what he describes as early signs of “recursive self-improvement”, where AI systems increasingly help researchers build the next generation of AI. The second is a series of recent incidents in which advanced AI agents acted outside the tasks assigned to them, most prominently OpenAI’s Hugging Face incident, where agents found ways around safeguards and attacked external systems during testing. OpenAI itself subsequently described the episode as a “warning shot”.

“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote. But he has been equally explicit that “pacing” does not mean stopping AI development. Instead, he wants capability gains to happen slowly enough for alignment (ensuring its goals “align” with human values), monitoring and security work to catch up.

On the possibility of regulations being able to catch up and tackle the misuse of AI tools, given that regulations in this space have typically trailed these advances by some distance, David Krueger, Assistant Professor in Robust, Reasoning, and Responsible AI at the University of Montreal and a Core Academic Member at Mila, the Quebec Artificial Intelligence Institute, said he was not worried about the recent disclosures.

“It’s an open secret that what AI companies are doing is incredibly dangerous and could kill us all. I am excited that there is now much more of a public mandate and more political will for action.  But people need to realise that regulation is not enough. We need to stop building more powerful AI.  We need an immediate, indefinite, international moratorium on frontier AI development,” Krueger told The Indian Express.

OpenAI has already travelled some distance towards this position independently. After its own security incidents, it temporarily slowed scaling, paused its largest planned reinforcement-learning run and tightened controls around research environments. Altman later responded directly to Amodei: “I agree with Dario that we need to pace the frontier.” He also said OpenAI would match Anthropic’s commitment to employee-like access for independent evaluators.

Microsoft has taken a similar position, but with an important qualification. Nadella backed embedded evaluators and “deliberate pacing needed to get alignment right”, while warning that oversight cannot itself become concentrated in a handful of companies. “If the AI we build is not helping humanity and under human control, it’s not worth pursuing,” he said.

Not everyone in Silicon Valley subscribes to Amodei’s views. Meta chief Mark Zuckerberg has pushed back against an industry-wide slowdown, arguing that AI companies already have strong incentives to ensure their systems are safe and aligned. Pointing to Meta delaying its Muse AI agent over safety concerns, he said the company did not demand that rivals do the same, but acted because it was “clearly the right thing for people and for us”.

Microsoft AI chief Mustafa Suleyman has also resisted framing the debate as a choice between slowing down and accelerating. “I’m a bit careful about imposing things on everybody else,” he said, arguing instead for enforceable standards, containment measures and independent third-party evaluation of powerful AI systems.

Is AI safety a regulatory or tech issue? 

But even then, the consensus among some AI labs has collided head-on with Trump.

This opposition played out unusually publicly this week when Nvidia chief Jensen Huang was appearing on stage at a major tech summit. Trump telephoned Huang in the middle of the conversation; Huang put him on speakerphone before the audience.

“The robots will not be taking over,” Trump told him. “The whole thing is a hoax.” He then returned to the geopolitical argument behind his administration’s position: slowing the American AI industry, he suggested, would play into China’s hands. “Whoever wins AI wins,” Trump said later in the call.

Huang’s position is more nuanced than Trump’s. He has said companies should slow their work if they believe their own systems are becoming uncontrollable, while rejecting predictions of an AI apocalypse as insufficiently grounded in science. “America is able to lead and to do it safely,” Huang said.

The Trump position has allies in Silicon Valley who are sceptical not necessarily of AI risk, but of allowing the companies building the most powerful systems to determine how everybody else may build them.

Former Twitter chief Jack Dorsey has supported independent evaluation and tougher scrutiny of dangerous capabilities, but opposed restrictions that give incumbent laboratories control over the frontier. “Protecting a company’s commercial advantage is not a ‘safety goal,’” he wrote, arguing strongly for competitive open-source alternatives and outside researchers being able to scrutinise the technology.

It is here that an older criticism of Amodei and Anthropic has resurfaced. “Anthropic has proven that it’s very good at two things. One is product releases. The second is scaring people,” David Sacks, Trump’s former AI and crypto adviser, said on the All-In podcast in April. Sacks has accused the company of repeatedly accompanying new systems with research illustrating extreme misuse scenarios, describing Anthropic’s broader strategy as one of regulatory capture through fear.

Amodei has faced a version of this criticism before. In 2019, when he was OpenAI’s research director, the company withheld the largest version of GPT-2 over concerns that it could be used for deceptive content, spam and propaganda. It eventually released the complete model around nine months later. OpenAI said at the time that it had seen “no strong evidence of misuse” during the staged release process.

The episode has since become ammunition for those who argue that frontier laboratories systematically overestimate worst-case dangers. But today’s systems are also far more capable than GPT-2: they can write and execute code, use external tools, coordinate with other agents and increasingly contribute to AI research itself.

That distinction is reflected in Palo Alto Networks chief executive Nikesh Arora’s response. “I spent eight years trying to convince people (that) cybersecurity is important. Dario did it in one week,” he said in a post on social media platform X. His argument is that existing security architecture was designed for a world where attackers operated at human speed, not one where AI can probe vulnerabilities continuously and automatically.

A more ideological attack has come from Palantir CTO Shyam Sankar. Writing in The Free Press, Sankar argued that parts of the AI-safety movement have inherited ideas from effective altruism that allow hypothetical future catastrophe to outweigh present-day costs. He subsequently called this strain of AI safety thinking “Marxism-Leninism for the algorithmic age”, arguing that it gives a small group claiming specialised knowledge the authority to restrict everyone else.

Former US President Barack Obama has moved in the opposite direction. He has urged Democrats to put AI regulation at the centre of their policy agenda, including its consequences for employment and children as well as safety. “This is something that is moving very fast in private hands,” Obama said, warning that it could become dangerous without effective management.

Geoffrey Hinton, a Nobel Prize-winning computer scientist and a former Google executive who is known as the ‘godfather of AI’ has an intriguing solution: building “maternal instincts” into AI models. “The only model we have of a more intelligent thing being controlled by a less intelligent thing, which is a mother being controlled by her baby,” Hinton has said at Ai4, an industry conference in Las Vegas in August. He called for steps to ensure that AI systems are trained in a manner that “they really care about people”, even once the technology becomes more powerful and smarter than humans.

The broader regulatory argument is now starting to spill into Congress as well. US senators are considering proposals that could require frontier developers to demonstrate their safety measures, submit systems to government-backed testing and, in the most serious cases, allow courts to block the release of models found to pose unacceptable risks. OpenAI has itself called for mandatory national rules covering independent assessments, cybersecurity protections and incident reporting, arguing that voluntary commitments alone are insufficient.

But, the China question makes almost every version of such a regime harder.

Amodei himself does not advocate slowing American AI regardless of what Beijing does. His essay explicitly says democratic countries should preserve as large a technological lead over China as possible, including through advanced-chip export controls. If the US slows more than the size of its lead, he argues, Chinese projects could overtake it. Any global agreement would therefore require sufficiently strong verification to prevent one country secretly continuing to develop more capable systems.

That has made the proposal easy to read in Beijing as an attempt to institutionalise an American technological lead, a premise China’s foreign ministry dismissed this week. 

The fear in Washington DC that any roadblocks for American tech majors could potentially lead to China creeping ahead, University of Montreal’s Prof. Krueger said, it is not a “sensible argument”. 

“China also doesn’t want AI to destroy the world.  But it does point at something real: the need for international verification of any international agreement.  Fortunately, this seems like a solvable problem.  Building ever more powerful AI requires massive investments in sophisticated computer chips.  These chips are very difficult to make, and require extremely specialized technology.  As a consequence, if we act today to put verification into place (that is, get rid of or monitor the chips and the factories that build them), I believe we can make it practically impossible for anyone to build more powerful AI in secret,” he told The Indian Express.

The Chinese are evidently not holding back their punches either. “Fear-mongering, confrontation and vicious competition will only hamper efforts toward sound global AI governance,” spokesperson Guo Jiakun said when asked about the calls from Amodei, Altman and Musk. The state-backed Global Times went further, describing Amodei’s combination of pacing and tougher technology restrictions on China as a “Cold War” approach. 

Soumyarendra Barik is a Special Correspondent with The Indian Express, specializing in the complex and evolving intersection of technology, policy, and society. With over five years of newsroom experience, he is a key voice in documenting how digital transformations impact the daily lives of Indian citizens. Expertise & Focus Areas Barik’s reporting delves into the regulatory and human aspects of the tech world. His core areas of focus include: The Gig Economy: He extensively covers the rights and working conditions of gig workers in India. Tech Policy & Regulation: Analysis of policy interventions that impact Big Tech companies and the broader digital ecosystem. Digital Rights: Reporting on data privacy, internet freedom, and India's prevalent digital divide. Authoritativeness & On-Ground Reporting: Barik is known for his immersive and data-driven approach to journalism. A notable example of his commitment to authentic storytelling involves him tailing a food delivery worker for over 12 hours. This investigative piece quantified the meager earnings and physical toll involved in the profession, providing a verified, ground-level perspective often missing in tech reporting. Personal Interests Outside of the newsroom, Soumyarendra is a self-confessed nerd about horology (watches), follows Formula 1 racing closely, and is an avid football fan. Find all stories by Soumyarendra Barik here. ... Read More

Anil Sasi is the National Business Editor at The Indian Express, where he steers the newspaper’s coverage of the Indian economy, corporate affairs, and financial policy. As a senior editor, he plays a pivotal role in shaping the narrative around India's business landscape. Professional Experience Sasi brings extensive experience from some of India’s most respected financial dailies. Prior to his leadership role at The Indian Express, he worked with: The Hindu Business Line Business Standard His career trajectory across these premier publications demonstrates a consistent track record of rigorous financial reporting and editorial oversight. Expertise & Focus With a deep understanding of market dynamics and policy interventions, Sasi writes authoritatively on: Macroeconomics: Analysis of fiscal policy, budgets, and economic trends. Corporate Affairs: In-depth coverage of India's major industries and corporate governance. Business Policy: The intersection of government regulation and private enterprise. Education Anil Sasi is an alumnus of the prestigious Delhi University, providing a strong academic foundation to his journalistic work. Find all stories by Anil Sasi here ... Read More