跳到正文
Rohan Paul· @rohanpaul_ai · X·· 3 小时前AI 评分51
AI 导读

Google 与卡内基梅隆的新论文发现,把安全作为独立目标演化时,自我改进智能体能更好地保持安全增益。智能体框架(模型周围的提示词、工具与循环)已能根据反馈自我重写,但多数系统只追逐单一分数如测试通过率,获得的安全增益在新任务上大多消失。

正文

New Google and Carnegie Mellon paper finds self-improving agents hold on to security gains far better when security is evolved as a separate goal.

Agent harnesses, the prompts, tools and loops around a model, can now rewrite themselves from feedback. Most such systems chase a single score, like test pass rate, and the security gains they pick up mostly vanish on new tasks.

Harmony Harness evolves several harness versions, each pushed toward a different balance of passing tests and blocking exploits. It also keeps general fixes separate from benchmark-specific ones, and picks which pieces to use per task.

On unseen crash types in SecRepoBench, 66.7% of its fixes both worked and blocked the exploit, against 36.7% for the starting harness. It kept 55% of its security gain on those tasks, while the DGM baseline kept 0%.

When you let an agent improve itself, track security as its own score and check it on held-out tasks.

来源:Rohan Paul · x.com