Anthropic researcher details AI system that improves model alignment

A researcher from Anthropic's fellows program has provided an early look at an AI system designed to improve other AI models' performance on alignment benchmarks. The paper, titled “Automated Researchers Can Reliably Mitigate Alignment Failures,” details how these automated systems can enhance a model's performance across various benchmarks without negatively impacting overall capabilities.

The system, led by Anthropic fellow Chen Yueh-Han, mimics traditional research by searching literature, proposing methods, and training models. It iteratively refines its approach, retaining effective methods and discarding ineffective ones, operating at scale and speed. The paper suggests that this automated post-training alignment could become practical in the near future, representing a significant step toward recursive self-improvement in AI.

According to the paper, the best Automated Alignment Researcher (AAR) method surpasses experienced human proposals, achieving this within approximately six hours. The cost-effectiveness is also highlighted, with AAR estimated at $4 per hour in API inference costs, significantly lower than the $150 per hour paid to human researchers. However, the paper acknowledges limitations, including the system's dependence on the accuracy of established benchmarks and the ongoing need to maintain and expand the literature base it draws from.