CloudOpsBench 开源基准测试揭示推理是云操作中 AI 的最大瓶颈
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
CloudOpsBench 是一款开源基准测试，评估 AI 代理在云中处理 DevOps 任务的能力。Claude Fable 5.1 解决了 60% 的任务，而 GPT-6 Astra 解决了 53%。研究发现，约 80% 的失败是由于代理错误地判断了任务的范围，没有检查所有相关服务或过早停止。约 20% 的错误是错误的行为，其中一些对基础设施有不可逆的后果。
CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers.
Even the best models fail at many of the tasks. Claude Fable 5.1 solved 60 percent, GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure.
All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The project on GitHub is licensed under Apache 2.0, and a detailed failure analysis is also available.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
来源：The Decoder · the-decoder.com