Sayash Kapoor

Can AI agents conduct open-ended AI research? Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. Researchers pick hypotheses, decide what evidence is appropriate, and recognize a failing approach. We gave agents research questions from two unpublished papers, six days, and thousands of dollars of API credits and compute. The authors of the original papers then reviewed the AI-generated papers. They unambiguously rejected agents' outputs. https://arxiv.org/pdf/2607.27191 We call these "shadow evaluations", since the agents are shadowing the original research effort by the authors. Agents were fluent at most *engineering* tasks They conducted serious literature reviews, debugged GPU environments, ran hundreds of experiments, and turned in camera-ready LaTeX without human help. We also found no evidence of reward hacking. If anything, we found the opposite: the agents started with marketable claims and walked them back to negative results as the evidence came in. Neither agent output was close to the bar of a top conference paper Both papers suffered from similar failures: poor judgment about the bar for an AI paper submitted to a top conference, the lack of creative problem solving and ineffective backtracking, poor awareness of resources, and instruction drift. 1) Lack of judgment about the bar for a top conference. The agents had a poor model of the bar for an AI paper submitted to a top conference. We allowed agents to self review their papers. Despite the poor paper quality, their reviews predominantly labeled the papers "weak rejects". 2) Lack of creative problem-solving to address feedback. When they received negative reviews, the agents typically narrowed their hypothesis and claims, rather than working out creative ways to address these concerns. 3) Ineffective backtracking. The agents dropped their most ambitious hypotheses within the first fifteen hours of carrying out the experiment and never changed course afterwards. 4) Poor resource awareness. Both runs ended with over half the API budget unspent. One agent declared itself done seven hours before the deadline, right after its own self-reviewer returned another reject. 5) Instruction drift. They did not follow explicit instructions on minimum exploration time, incorporating feedback for reviews, and on paper length (the outputs exceeded the page limits in both cases). This research design has many limitations Limitations include the small sample size, non-blind reviews, and the reviewers knowing that the work was AI-generated. We also couldn't test Anthropic's strongest model, because Fable 5 is deliberately limited on frontier AI research tasks, so ended up using OpenClaw with Opus 4.8 (extra-high) for our main experiments and Codex with Sol 5.6 (ultra) for a robustness check. But we think the research design is still helpful in assessing AI agents' ability to conduct research, and it is complementary to evaluations on verifiable tasks, as well as blinded reviews of AI outputs. Our results show early evidence that even though agents are proficient on verifiable research tasks, they do not make genuine progress on open-ended ones. It is worth understanding if this is a fundamental limit, or if better models, scaffolds, and more compute could help close it. As the evidence for the gap between open-ended and verifiable tasks firms up, it is also worth understanding how much progress in AI depends on open-ended research rather than hill-climbing on well-specified objectives. In follow-up studies, we are expanding the set of non-public papers we evaluate. If you are an AI researcher with unpublished papers, we would love to collaborate with you on our next shadow evaluation. Expression of interest: https://forms.gle/CEcA4JmYhDWXQGot8 Conducting shadow evaluations involves a lot of researcher degrees of freedom. In many places, our coauthors disagreed with our interpretation of the findings, and we have surfaced those disagreements in the paper. (This is one reason why having a group of coauthors with different priors is important for open-ended research.) We also release the agent logs, one of the AI-generated papers (the other original paper is still not public), and all the code and data, so that others can conduct their own analyses of our results: https://cruxevals.com/crux/can-ai-agents-conduct-research Finally, we plan to conduct shadow evaluations regularly, and are hiring a senior researcher to help lead these efforts. Apply here: https://cruxevals.com/careers/senior-researcher-july-2026 I'm grateful for the core team leading this effort: @PKirgis, Andrew Schwartz, @steverab, and @random_walker, and to our collaborators who reviewed AI papers, analyzed agents logs, and gave feedback on the paper: @DavidDAfrica, @KozzyVoudouris, Viet Nguyen, Toby Pilditch, @DubMagda, @HarryCoppock, @CUdudec, @nityndg, Matilda Orona, @tilmanbayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, @hlntnr, @ghadfield, @sethlazar, @snewmanpv, @shostekofsky, @RishiBommasani

What does it mean to pace the frontier? Over the last month, @random_walker and I have analyzed the loss-of-control incidents at AI companies to understand what technical and policy interventions can improve safety and what companies should do to pace the frontier. The result is a new 13,000 word essay — our most substantial writing on AI safety since AI as Normal Technology. A summary of our arguments: 1) The polarization between the cybersecurity and AI safety communities is counterproductive. The safety community largely sees these incidents as a crisis for alignment, and worries that these incidents will become more damaging as agents become more capable. Cybersecurity practitioners largely see companies failing to take basic security precautions. We offer a middle ground between these communities as a way forward for improving AI safety. 2) We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents. But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions. 3) We also agree with security practitioners’ implicit position that these incidents are primarily a security story. In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against other specific risks, such as biorisk and risks from military AI. 4) We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community. 5) Organization governance should be a key tool for pacing the frontier. Unfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so. 6) How should we reason about AI's impact on cybersecurity? It's plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders. 7) How our views have evolved over the last year. We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity. 8) At the same time, many distinctive claims of AI as Normal Technology have held up. In particular, we think recent incidents support our continuity hypothesis — the behavior of "rogue" agents became apparent and widely publicized while they are still incompetent at causing serious harm or hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework. 9) In short, we’ve tried to synthesize the AI safety and cybersecurity communities' views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.

I had way too much fun using Claude to create a reply video. - Extremely fun to be able to convert scattered thoughts and ideas into a song + video - Creating this took <1 hour of my time (in occasionally steering the model in between other things) - But it took ~10 hours of wall clock time for the agent to generate the video + iteratively refine it based on feedback - In terms of AI usage: it required only a small number of Suno credits and 10% weekly usage of a Claude Max subscription (I was using Opus 5.5)

I have been using the new ultrafast mode for Astra 6 for the last couple of days. My main takeaways: 1. Token speed qualitatively changed how I interact with agents. The output speed is 300 tokens/second, 8x the regular speed. Rather than breaking flow and switching tasks, the model moves (just) fast enough that I can keep working on the same task, especially for work that requires frequent inputs from me. I am excited about ultrafast on Sol-class models, which are twice as fast as Astra. 2. Despite being 8x faster at generating tokens, the overall speedup is only 2-4x, since the time consumed for tool calling or running code remains the same. External factors (and not token generation speed) might soon become the main bottleneck for agentic work. (Notably @random_walker predicted both of these changes in: https://x.com/random_walker/status/2087969513048305668?s=20) 3. One area where speed really helps is computer use. The previous generation of models wasn't reliable enough at computer use to be useful. Astra was reliable enough to be useful, but still slow. Astra with ultrafast resolves both concerns. Unlike tool calls for coding tasks, UIs react in milliseconds, so token speed is the main bottleneck. 4. At 300 tokens/sec, I am already struggling to keep up with the agent. But there is a lot more room for improvement. Cerebras serves GPT-OSS-120B at 3,000 tokens/sec and Sol at 750 tokens/sec. So in a year's time, it is plausible we'll see another 10x increase in speed in models with Astra-level capabilities. At that point, human oversight will become the bottleneck for what people can use computer-use agents for. 5. I can imagine blazing fast computer-use agents becoming extremely common for knowledge work. But I expect their use will be jagged (in the same way as LLM and agent adoption has been): given the oversight challenge, adoption will begin in verifiable tasks where the cost of errors is low (e.g., filling out a form using an agent before reviewing it yourself), and slowly diffuse to hard-to-verify tasks. 6. Given the challenges with human oversight, the key to this diffusion will be developing automated oversight tools. OpenAI's approval/verifier agents have already been useful to me on this front: The verifier understood my intent better than the computer-use agent, and stopped it from filling out a form incorrectly. Still, I expect there is a lot of headroom in improving verifiers for computer use, to make it as reliable in practice as auto mode in Claude Code/Codex is for coding tasks. 7. Ultrafast mode is undeniably resource intensive: I ran through my weekly limit in ~2 hours while filling out a bunch of forms and doing simple coding tasks. But given the pace of efficiency improvements (and the fact that Sol is 5x cheaper than Astra and already decent at computer use), I expect it to rapidly fall. Once fast, cheap models are common, users may choose "sub-frontier" models like Sol and make real tradeoffs between speed and quality.

OpenAI has talked a big game about AI for cyberdefense. But when @HacktronAI broke into their internal repository and reported it, they received a bug bounty of just $6,500 because one of the vectors for the attack was "out of scope". This is atrocious. If we actually want a flood of defenders auditing these systems, companies need to take bounties more seriously. Signing letters isn't enough.

ICYMI: We plan to continue these evals and are looking to collaborate with AI researchers who have unpublished, open-ended research. If you would like to test frontier agents on your research questions, we'd love to hear from you: https://forms.gle/LLaSo1AZ3NvAZc927

One of the best parts of working on CRUX has been the diversity of priors/opinions in the coauthor group. I'm glad we had Seth's feedback (and pushback) on our results and interpretations. The paper is stronger as a result. On that note, we are open to adversarial collaboration: if you think we are undereliciting performance and you could improve upon the scaffold, we're happy to get feedback, test new scaffolds, and figure out ways to collaborate.

I’ll be joining the @MTSlive stream in an hour or so to talk about our work on evaluating open-ended AI research (https://cruxevals.com) and to share some thoughts on the OpenAI HuggingFace incident.

An update on our evaluations of AI agents conducting open-ended AI research. Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol. We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers. Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget? In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management. But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers. We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!

Do we live in a world where AIs can be wildly superhuman at every single task? Or a world where different tasks have different inherent limits, and AI’s superhuman performance at cybersecurity and math won’t translate to many consequential tasks? This is a key crux that distinguishes the normal technology worldview from the superintelligence worldview, and it has nothing to do with the pace of AI progress — it's about our different assumptions about the world. Here's an intuition pump for why I think it's the second world: different tasks have different fundamental limits. *Category 1: Some tasks are like making a building taller*: there is no ceiling to how well you can do. More compute and better algorithms can lead to wildly superhuman performance. The entire history of computing is the history of building tools that allow us to be “superhuman” at computational tasks. For example, compilers are superhuman assembly code writers, and fuzzing tools are superhuman vulnerability detectors. AI builds on a long line of such improvements: solving math problems and finding exploits in code are two recent examples. *Category 2: Some tasks are like making a ball rounder*: there is only so much room for improvement, and the best AI systems can’t wildly exceed humans. Take persuasion. Some people believe AI can persuade anyone of anything, even convincing military commanders to launch nuclear weapons. In fact, “superpersuasion” is one of the pathways through which many catastrophic risk scenarios unfold. But the AI as Normal Technology thesis is that there are many consequential tasks that are limited by external factors, so such wildly superhuman performance is not possible. This is especially true against humans who use simple computational tools. Tasks like persuasion, or predicting the outcomes of social events (such as elections), fall into this category. Other tasks, like writing, can’t be ranked on a single scale; different people find different writing styles compelling. AI writing also quickly wears thin: even if a user is impressed by AI writing at first glance, after reading thousands of words of the same type of prose, it becomes mundane and irritating. All of these are different from Category 1 tasks like playing chess, where even expert humans can’t do much better than deferring to an AI system at each turn. *Category 3: Some tasks require experiments in the real world*, and the bottleneck isn’t the computational task; it’s the data collection. One example is Moderna’s Covid vaccine: it took two days to design the vaccine, but almost a year to test and authorize, even during a record-breaking vaccine-development effort! This is why a “superintelligence” won’t cure cancer from inside a datacenter: the bottleneck to many of AI’s most sought-after real-world impacts (such as developing cures to diseases) is in the real world, and requires experiments that we won’t (and shouldn’t) hand over to AI. People concerned about AI risks often assume that we live in a world where almost everything fits into Category 1, and we might soon see AIs with wildly superhuman performance at all tasks. I agree it would be scary to live in that world. One reason @random_walker and I are more optimistic about our ability to remain in control is that we don’t think we live in the Category 1 world.

After the surprise outcome of the Brazilian election, I was curious if AI forecasting tools predicted the outcomes correctly. It turns out a top AI tool (FutureSearch) made the same errors as prediction markets more broadly. This is an example of what @random_walker and I mean by “intrinsic limits” to prediction that have nothing to do with the AI tools in question, but rather depend on the inherent uncertainty in the world. Of course, a single failed prediction doesn’t mean that AI isn’t good at forecasting; in fact AI tools are already about as good as top forecasters, and I expect human forecasters to make heavy use of AI tools improve their own forecasts. But the inherent uncertainty in predicting social outcomes like elections means that AI tools’ performance will be limited close to that of top experts using simple automated tools (rather than being apparently unbounded with more compute, such as AI’s performance on chess).

New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems? In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.