Anthropic pushes for global AI safety mechanisms

Anthropic said on Thursday that developers of advanced artificial intelligence systems should establish a coordinated and verifiable framework to slow down or temporarily halt development if future AI models begin improving themselves at a pace that exceeds society’s ability to manage the associated risks, News.Az reports, citing Reuters.

The AI company noted that systems capable of building improved versions of themselves would represent a major technological milestone. However, it warned that “full recursive self-improvement” could also heighten the risk of humans losing control over increasingly powerful AI systems.

“If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important,” the company said.

As an illustration of AI’s growing role in software development, Anthropic stated that, as of May, more than 80% of the code merged into its codebase had been written by its AI assistant, Claude.

According to the company, it would be beneficial for society to retain the option of slowing or temporarily pausing frontier AI development, allowing governance structures, safety measures and alignment research to keep pace with technological advances.

At the same time, Anthropic cautioned that unilateral or poorly coordinated slowdowns could prove counterproductive if other, less cautious developers continue advancing the technology. Such a scenario, the company argued, could ultimately reduce overall safety rather than improve it.

The firm emphasized that any meaningful pause would require the participation and agreement of several well-resourced AI laboratories operating at the technological frontier. It also said clear rules would be needed to determine what conditions would trigger or end a pause, as well as who would be responsible for overseeing the process.

While a single company could more easily implement a unilateral pause, Anthropic said such a move would likely have limited impact, mainly shifting industry leadership rather than encouraging broader international discussion and coordination.

Anthropic’s research division, the Anthropic Institute, plans to examine and help develop the systems and frameworks that would be necessary to support a coordinated slowdown mechanism.

Over the coming months, the company intends to bring together policymakers, researchers, civil society organizations and other AI firms to discuss critical issues related to advanced AI development.

Among the topics to be explored are methods for managing risks associated with recursive self-improvement and ways to strengthen coordination mechanisms among developers and stakeholders.

Last month, Anthropic completed a fundraising round that valued the company at $96.5 billion. The company also confidentially filed for an initial public offering in the United States on Monday.

News.Az 

By Nijat Babayev