What are we betting on? Assessing the state of macrostrategy in AI safety as timelines shorten
Summary
- AI R&D is accelerating. But it's still hard to say what the AI safety field, considered as a portfolio, is betting on to make AI go well. What are the key assumptions that organizations are making, and where are there gaps that could leave us vulnerable to unexpected futures? These questions are even more urgent as timelines shorten.
- With support from BlueDot Impact, I'm launching a macrostrategy project. I plan to:
- map the theories of change of 15-20 key AI safety organizations against field-wide theories of victory
- surface shared assumptions (including the extent to which projects and orgs' success depends on longer timelines, or a pause of some kind - or not, as it may be)
- identify signals we could use to test those assumptions and course correct
- and pick out gaps where we need more investment and resources.
- By February 2027, I intend to have a rough snapshot of the overall strategic landscape to share, with a full report and recommendations for funders and field strategists ready in April/May. I'll also provide short updates along the way at mikeyjm.com.
- I would love your input: feedback, advice, strategy documents, an interview (all data will be anonymized in the outputs), or more generally, thoughts about how to make this project as useful as possible (or whether it's not useful in the first place!).
The problem
AI capabilities are accelerating. In early November, METR found that the pace of progress on well-defined tasks at Anthropic has roughly doubled, with AI accounting for roughly 80% of that doubling. Then, on November 10, Anthropic updated its S-1 filing with the SEC, noting that it has already met its own Responsible Scaling Policy's threshold for AI-accelerated AI R&D (based on what it expects from future models). And in its IPO roadshow, the company told investors that Claude will be able to find compute improvements "entirely by itself." If the current pace continues (or accelerates), many forecasters and researchers now expect to see RSI in the next 6-12 months, with superhuman AI researchers and/or superintelligence following soon after (see, for example, Ajeya Cotra, the AI Futures Project, Buck Shlegeris, and Nikola Jurkovic).1
How should the AI safety field respond? Where should we put our efforts, given how much uncertainty, and how many big open questions (for example, about timelines, the offense-defense balance, whether alignment is likely to scale, etc.), we're confronting? How do we make field-level bets that are likely to pay off in the short term without giving up the work needed to make AI go well in the medium and longer term - especially when some of the most important projects (long time horizon alignment work, for example) can't really be scoped down to be useful in a short timelines world?
I don't see self-evidently correct answers here. And that's exactly why field-level macrostrategy matters urgently right now. If we can see the hundreds of AI safety organizations and projects as a portfolio, in the aggregate, we can identify the assumptions that elements of that portfolio depend on, test those assumptions, and use emerging evidence to allocate resources even more effectively. For the field to hold up, no matter what the future brings, we need to make sure that we're hedging our bets at portfolio level.
As far as I can tell, we don't yet have the sort of field-level view of the overall AI safety portfolio that would enable us to know whether and to what extent we're hedging our bets. There are good field maps (e.g., Harry Waterman's, AISafety.com's, and Morgan Matthews' AI Safety Landscape), great strategic analyses (e.g., Adam Jones, Jason Hausenloy, Oscar Delaney, and Ryan Greenblatt), and I've spoken to at least a couple of organizations building out their own landscape scans. But there still isn't a clear, systematic picture of how AI safety organizations - in various domains - and their strategies line up with field-wide theories of victory, what assumptions they share, and where gaps in the overall portfolio are (at least not publicly - government AISIs may be trying to ramp up coordination).
My working hypothesis (which I previously articulated in How Would We Know) has three parts:
- The macrostrategic core of AI safety is too small. By core, I mean strategies that pay off in most futures, such as investments in evidence and verification, strategic coordination, and building political will.
- Many projects essentially function as a set of somewhat disconnected bets, without much connective tissue between them.
- Very little attention is paid to shared assumptions, or to specifying and tracking signals that would tell us when those assumptions are failing, and figuring out how to change course.
For example, many people seem to be increasingly in favor of a pause on AI development, in some form, but - despite lots of really great work on compute verification (e.g., Tom Milton on verifying a US-China deal), political advocacy (e.g., the call for an international AI treaty), and strategic coordination (e.g., MIRI's proposed international agreement; even the MAIM deterrence framework, loosely considered, would fit in this bucket) - there still seems to be far too little attention to how, exactly, to make a pause workable, especially in the very near future (see box).
Why now? And why not just push for a pause?
If RSI is just months away, isn't macrostrategy hopelessly meta? Shouldn't we instead put all of our efforts into securing a pause? I think these are fair questions. I would agree that a lot of effort should go into securing a pause, and into working out how to implement one that actually reduces risk, domestically and eventually globally, rather than simply delaying it or making it worse over time (Belrose aptly describes some of the ways a pause could backfire).
But I see deciding how hard to push for a pause as a portfolio question. Calls to shift the field toward short-timelines work are already portfolio-level claims: they assume that too much of the AI safety work being done now only pays off on longer time horizons. I have a lot of sympathy for Nikola Jurkovic's recent post on LessWrong - but I think we need better evidence for assessing who is working on what, and under what assumptions, in AI safety. Right now, I don't think we can credibly say how much of the work being done across the field assumes short timelines, how much depends on a pause happening or not, or what assumptions might fail first. In other words, we need to know what the portfolio is before we can decide how to (re)calibrate it.
So that's what this project aims to provide - quickly! If we have a rough map of the strategic landscape, we can then move resources across the portfolio to where they're most needed now. A fuller analysis could show, for example, what a pause would actually require, who is positioned to make it happen, and what else would need to be true for one to reduce risk.
With support from a BlueDot Impact Career Transition Grant, I'm launching a project to test this hypothesis.
The plan
Starting in mid-December, I intend to explore:
- how key AI safety organizations - field builders, technical researchers, technical governance orgs, grantmakers, futurists, and more - expect their work to help AI go well (in other words, what their theories of change are, whether they're written down or not)
- how those theories of change line up (or don't) with common, field-wide theories of victory (defined by Hausenloy, roughly, as a full causal account of the steps that need to go right, and a plan for achieving them at field-level)
- how much of that work assumes short timelines, and/or depends on a pause of some sort happening, or not
- which assumptions organizations share, what signals we could use to test those assumptions in the very near future, and where the overall portfolio has gaps
To be clear, I don't think the field needs a single, collectively shared theory of victory. But I do think we'd do well to have a better sense as to which theories of victory we're actually betting on, whether we're doing the work needed to advance them, and whether those theories make object-level sense in the first place.
The goal here is to produce practical recommendations on field-wide macrostrategy that funders and field strategists can use to make decisions about their own portfolios, helping the field as a whole be more robust to multiple futures - including futures in which RSI happens very soon. My idea is that if we can make the macrostrategic landscape visible, we can then make it even stronger together.
How I'll do this
I'll work bottom-up and top-down at the same time:
- From the bottom: I'll identify a sample of 15-20 key organizations across different domains and functions, and work to specify their theories of change, including explicit and implicit assumptions. I'll do this by gathering, reviewing, and analyzing existing strategy documents and other reports, and by conducting semi-structured interviews with staff and leadership about what they do, in practice, and why.
- From the top: I'll start from existing accounts of field-wide theories of victory (e.g., Hausenloy, Delaney), and refine them as I get evidence about how orgs think about making AI go well overall.
Then I'll map the theories of change to the theories of victory, and systematically draw out shared assumptions, gaps, and signals that need tracking so that we know what's working, what's not, and how to adjust.
My plan isn't to build a new AI safety field map (though I should say, my favorite one of these remains Morgan Matthews' AI Safety Landscape, which maps strategies from threat models through to theories of victory). Instead, my analysis will add strategic clarity about what organizations are actually doing and assuming, the signals that would tell us whether their assumptions are holding, and how it all fits together as a portfolio. Building and evaluating theories of change - at the organizational and portfolio levels - is the core of what I've spent my career doing, and that's why I think I can add some value here.
As I go, I will share emerging findings and questions on a regular basis, both to keep the field informed, and to make sure I'm getting feedback to steer this project in the most useful direction possible. I won't be judging or reporting on any particular organization and its theory of change - all findings will be reported by domain, anonymized and/or aggregated, and my focus will be on making sense of the field as a whole.
I will manage this project adaptively. If the evidence suggests that acceleration continues - with, for example, OpenAI or another lab reporting the same rate of acceleration as Anthropic, or AI systems autonomously running R&D - or if there's a serious incident involving current models, I'll reframe the project to focus in on what matters most at that point: like how we get a safe, sustainable, and effective pause in place, including what else is required to make it possible in the next 6-12 months.
Timeline and deliverables
- December: a full research plan and sample of orgs to engage
- January: completed document review
- February: rough portfolio snapshot, in which I share emerging results from the document review, including shared assumptions and gaps, with particular attention to timelines and the extent to which payoff depends on a pause. My thought is that this will be useful in and of itself, and give me something to test and refine as I conduct interviews. I will also kick off interviews.
- March: interviews complete
- April/May: a full interactive report, updated to include evidence from the interviews, setting out the findings and recommendations for strengthening macrostrategy across the field, including assumptions, signals, and tripwires - including analysis of where the portfolio is thin, and how funders might bolster it
Throughout, I'll post short updates every few weeks at mikeyjm.com, sharing what I'm up to, and early findings and questions. I'll also make at least a few posts on LessWrong, including the interim snapshot and a short version of the final report.
Who is this for?
The audience for this work is funders and grantmakers, macrostrategists, and staff at AI safety orgs who are trying to make sure AI goes well. All of these people (all of you!) already do their own strategy work, and this is intended to complement it. What this project aims to contribute is a field-level analysis that would be hard for a single org to produce on its own, with a clear, evidence-based perspective on patterns and shared assumptions and gaps that are hard to pull out of ad hoc bilateral conversations or exchanges.
How you can help
- Tell me what you think. How could this project be most useful to you and your colleagues? Email me at moses@mikeyjm.com, or book a time to talk about it.
- Share how your organization thinks about making AI go well. Send me any strategy documents and/or progress reports you're able to share, or let me know if you'd be willing to do a 60-minute interview in February or March. To be clear, I won't publicly attribute anything to your organization (see text above).
- Join a small advisory group that I'll check in with for 30 minutes each month, to give feedback on the design and on emerging findings.
- Tell me who I should talk to, and make introductions if you're up for it.
- Push back. If you think the framing isn't helpful, or the project duplicates existing work, or this is just generally a bad idea, I want to know!
Like these folks, I'm pretty concerned about the recent evidence regarding capabilities progress. I've brought my own timelines for RSI forward (see below). But I continue to think that RSI is not the most useful frame for thinking about timelines - people often mean different things when they use the term, and it's not clear to me that RSI, however construed, will automatically eliminate all the bottlenecks that might constrain an intelligence explosion (for example, research taste (Ball and Huang); sample efficiency (Dwarkesh - though Dwarkesh now thinks this is more tractable than he used to); whether returns to software research hold up once compute is fixed (Eth & Davidson - though I should say, these guys think an explosion is fairly likely even despite this constraint; Huang also sees compute as a constraint); and whether capability gains will indefinitely scale and generalize beyond verifiable domains like math and software engineering). I take Yudkowsky's point, via Paul Christiano, in this comment thread, that "intelligence is magic." But the METR report is limited to "well-specified tasks" - there is some kind of magic happening, but whether it cascades to less well-specified domains, and at what rate, remains unclear to me.
That said, I have updated since How Would We Know, in large part due to the METR report and the S-1. In October, I put a ~30-40% chance on transformative AI in the next 1-3 years; I'd now put it at roughly 50-55%, with an outside chance it comes sooner. That makes it even more important that we treat TAI as imminent, and that our macrostrategy - considering the field as a portfolio - seeks to both minimize short-term harms and maximize long-term payoffs. This doesn't mean that every single project needs to balance these things; it means that, at the portfolio level, we need to have our bases covered.↩