AI Post Transformers podcast

ASI-Bench: Measuring AI Agents' Scientific Autonomy Without Method Hints

0:00
NaN:NaN:NaN
Retroceder 15 segundos
Avanzar 15 segundos
This episode examines ASI-Bench, a benchmark that tests how far AI research agents can carry a scientific project on their own as human methodological guidance is progressively removed. It covers the four-level "hint ladder" (full method, method-level guidance, objective-only, and objective plus irrelevant distractors) applied to the same 60 multi-stage tasks across 11 domains. Agent-and-model pairs are evaluated, and the harness is shown to change scores for the same model. It contrasts this design with MLE-bench, RE-Bench, PaperBench and ScienceAgentBench, which fix or only coarsely vary how much method the agent is given. The discussion centers on the authors' claim that turning a named approach into a working procedure, rather than choosing the approach, is the main bottleneck. It also weighs a competing reading: the score drop when hints disappear may partly reflect lost information such as exact equations and constants, not just a failure of operationalization. Listeners get a skeptical look at what the benchmark's construction (over 40 experts, 31,000 human hours, leakage checks) can and cannot establish about agent autonomy. Sources: 1. ASI-Bench: At the Dawn of Artificial Superintelligence — Junwei Zhou, Zhen Sun, Binyu Li, Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Xiaoyu Cao, Yongchao Chen, Yuanning Feng, Junhao Wu, Cheng Zhang, Sijia Chen, Haoyu Xue, Chengsong You, Huan Wang, Koutian Wu, Peigan Gao, Jiakun Wu, Wenzhe Li, Ergan Shang, Qingyuan Zheng, Jingjing Zhou, Ruixuan Jia, Yan Xu, Hongrui Zhang, Xiao-Han Ma, Zhengxiang Cheng, Yuexing Hao, Liting Mai, Xianglin Ji, Wenjun Zhang, Zhuofan Chen, Yixiao Huang, Chi Wang, Wenyue Hua, Yilun Hao, Yuantao Zhai, Ziyan Zhao, Jingyan Xie, 2026 http://arxiv.org/abs/2608.17271 2. ASI-Bench: At the Dawn of Artificial Superintelligence — Junwei Zhou, Zhen Sun, et al.; corresponding author Yongchao Chen (Tsinghua University and collaborators), 2026 https://scholar.google.com/scholar?q=ASI-Bench%3A+At+the+Dawn+of+Artificial+Superintelligence 3. PaperBench: Evaluating AI's Ability to Replicate AI Research — Giulio Starace, Oliver Jaffe, et al. (OpenAI), 2025 https://scholar.google.com/scholar?q=PaperBench%3A+Evaluating+AI%27s+Ability+to+Replicate+AI+Research 4. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — Jun Shern Chan, Neil Chowdhury, et al. (OpenAI), 2024 https://scholar.google.com/scholar?q=MLE-bench%3A+Evaluating+Machine+Learning+Agents+on+Machine+Learning+Engineering 5. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Carlos E. Jimenez, John Yang, et al. (Princeton), 2023 https://scholar.google.com/scholar?q=SWE-bench%3A+Can+Language+Models+Resolve+Real-World+GitHub+Issues%3F 6. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery — Ziru Chen et al., 2024 https://scholar.google.com/scholar?q=ScienceAgentBench%3A+Toward+Rigorous+Assessment+of+Language+Agents+for+Data-Driven+Scientific+Discovery 7. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts — Hjalmar Wijk et al., 2024 https://scholar.google.com/scholar?q=RE-Bench%3A+Evaluating+frontier+AI+R%26D+capabilities+of+language+model+agents+against+human+experts 8. AI Scientist (v1/v2) — Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada et al., 2024/2025 https://scholar.google.com/scholar?q=AI+Scientist+%28v1%2Fv2%29 9. ProjectionBench: Evaluating scientific hypothesis generation in LLMs under progressive information disclosure — A. Lew, Y. Cao, M. J. Buehler, 2026 https://scholar.google.com/scholar?q=ProjectionBench%3A+Evaluating+scientific+hypothesis+generation+in+LLMs+under+progressive+information+disclosure 10. ProjectEval: A benchmark for programming agents automated evaluation on project-level code generation — Kaiyuan Liu et al., 2025 https://scholar.google.com/scholar?q=ProjectEval%3A+A+benchmark+for+programming+agents+automated+evaluation+on+project-level+code+generation 11. FIRE-Bench / ResearchClawBench — Zhen Wang et al.; Wanghan Xu et al., 2026 https://scholar.google.com/scholar?q=FIRE-Bench+%2F+ResearchClawBench 12. SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks? — Xiang Deng et al., 2025 https://scholar.google.com/scholar?q=SWE-bench+Pro%3A+Can+AI+agents+solve+long-horizon+software+engineering+tasks%3F 13. Chain-of-thought and prompting ablations on information level (e.g. few-shot versus zero-shot, and Wei et al. 'Chain-of-Thought Prompting Elicits Reasoning') — Jason Wei et al., 2022 https://scholar.google.com/scholar?q=Chain-of-thought+and+prompting+ablations+on+information+level+%28e.g.+few-shot+versus+zero-shot%2C+and+Wei+et+al.+%27Chain-of-Thought+Prompting+Elicits+Reasoning%27%29 14. Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi et al., 2023 https://scholar.google.com/scholar?q=Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context Interactive Visualization: ASI-Bench: Measuring AI Agents' Scientific Autonomy Without Method Hints

Otros episodios de "AI Post Transformers"