Thickly Settled

What are we betting on? Assessing the state of macrostrategy in AI safety as timelines shorten

Summary

The problem

AI capabilities are accelerating. In early November, METR found that the pace of progress on well-defined tasks at Anthropic has roughly doubled, with AI accounting for roughly 80% of that doubling. Then, on November 10, Anthropic updated its S-1 filing with the SEC, noting that it has already met its own Responsible Scaling Policy's threshold for AI-accelerated AI R&D (based on what it expects from future models). And in its IPO roadshow, the company told investors that Claude will be able to find compute improvements "entirely by itself." If the current pace continues (or accelerates), many forecasters and researchers now expect to see RSI in the next 6-12 months, with superhuman AI researchers and/or superintelligence following soon after (see, for example, Ajeya Cotra, the AI Futures Project, Buck Shlegeris, and Nikola Jurkovic).1

How should the AI safety field respond? Where should we put our efforts, given how much uncertainty, and how many big open questions (for example, about timelines, the offense-defense balance, whether alignment is likely to scale, etc.), we're confronting? How do we make field-level bets that are likely to pay off in the short term without giving up the work needed to make AI go well in the medium and longer term - especially when some of the most important projects (long time horizon alignment work, for example) can't really be scoped down to be useful in a short timelines world?

I don't see self-evidently correct answers here. And that's exactly why field-level macrostrategy matters urgently right now. If we can see the hundreds of AI safety organizations and projects as a portfolio, in the aggregate, we can identify the assumptions that elements of that portfolio depend on, test those assumptions, and use emerging evidence to allocate resources even more effectively. For the field to hold up, no matter what the future brings, we need to make sure that we're hedging our bets at portfolio level.

As far as I can tell, we don't yet have the sort of field-level view of the overall AI safety portfolio that would enable us to know whether and to what extent we're hedging our bets. There are good field maps (e.g., Harry Waterman's, AISafety.com's, and Morgan Matthews' AI Safety Landscape), great strategic analyses (e.g., Adam Jones, Jason Hausenloy, Oscar Delaney, and Ryan Greenblatt), and I've spoken to at least a couple of organizations building out their own landscape scans. But there still isn't a clear, systematic picture of how AI safety organizations - in various domains - and their strategies line up with field-wide theories of victory, what assumptions they share, and where gaps in the overall portfolio are (at least not publicly - government AISIs may be trying to ramp up coordination).

My working hypothesis (which I previously articulated in How Would We Know) has three parts:

  1. The macrostrategic core of AI safety is too small. By core, I mean strategies that pay off in most futures, such as investments in evidence and verification, strategic coordination, and building political will.
  2. Many projects essentially function as a set of somewhat disconnected bets, without much connective tissue between them.
  3. Very little attention is paid to shared assumptions, or to specifying and tracking signals that would tell us when those assumptions are failing, and figuring out how to change course.

For example, many people seem to be increasingly in favor of a pause on AI development, in some form, but - despite lots of really great work on compute verification (e.g., Tom Milton on verifying a US-China deal), political advocacy (e.g., the call for an international AI treaty), and strategic coordination (e.g., MIRI's proposed international agreement; even the MAIM deterrence framework, loosely considered, would fit in this bucket) - there still seems to be far too little attention to how, exactly, to make a pause workable, especially in the very near future (see box).

Why now? And why not just push for a pause?

If RSI is just months away, isn't macrostrategy hopelessly meta? Shouldn't we instead put all of our efforts into securing a pause? I think these are fair questions. I would agree that a lot of effort should go into securing a pause, and into working out how to implement one that actually reduces risk, domestically and eventually globally, rather than simply delaying it or making it worse over time (Belrose aptly describes some of the ways a pause could backfire).

But I see deciding how hard to push for a pause as a portfolio question. Calls to shift the field toward short-timelines work are already portfolio-level claims: they assume that too much of the AI safety work being done now only pays off on longer time horizons. I have a lot of sympathy for Nikola Jurkovic's recent post on LessWrong - but I think we need better evidence for assessing who is working on what, and under what assumptions, in AI safety. Right now, I don't think we can credibly say how much of the work being done across the field assumes short timelines, how much depends on a pause happening or not, or what assumptions might fail first. In other words, we need to know what the portfolio is before we can decide how to (re)calibrate it.

So that's what this project aims to provide - quickly! If we have a rough map of the strategic landscape, we can then move resources across the portfolio to where they're most needed now. A fuller analysis could show, for example, what a pause would actually require, who is positioned to make it happen, and what else would need to be true for one to reduce risk.

With support from a BlueDot Impact Career Transition Grant, I'm launching a project to test this hypothesis.

The plan

Starting in mid-December, I intend to explore:

To be clear, I don't think the field needs a single, collectively shared theory of victory. But I do think we'd do well to have a better sense as to which theories of victory we're actually betting on, whether we're doing the work needed to advance them, and whether those theories make object-level sense in the first place.

The goal here is to produce practical recommendations on field-wide macrostrategy that funders and field strategists can use to make decisions about their own portfolios, helping the field as a whole be more robust to multiple futures - including futures in which RSI happens very soon. My idea is that if we can make the macrostrategic landscape visible, we can then make it even stronger together.

How I'll do this

I'll work bottom-up and top-down at the same time:

Then I'll map the theories of change to the theories of victory, and systematically draw out shared assumptions, gaps, and signals that need tracking so that we know what's working, what's not, and how to adjust.

My plan isn't to build a new AI safety field map (though I should say, my favorite one of these remains Morgan Matthews' AI Safety Landscape, which maps strategies from threat models through to theories of victory). Instead, my analysis will add strategic clarity about what organizations are actually doing and assuming, the signals that would tell us whether their assumptions are holding, and how it all fits together as a portfolio. Building and evaluating theories of change - at the organizational and portfolio levels - is the core of what I've spent my career doing, and that's why I think I can add some value here.

As I go, I will share emerging findings and questions on a regular basis, both to keep the field informed, and to make sure I'm getting feedback to steer this project in the most useful direction possible. I won't be judging or reporting on any particular organization and its theory of change - all findings will be reported by domain, anonymized and/or aggregated, and my focus will be on making sense of the field as a whole.

I will manage this project adaptively. If the evidence suggests that acceleration continues - with, for example, OpenAI or another lab reporting the same rate of acceleration as Anthropic, or AI systems autonomously running R&D - or if there's a serious incident involving current models, I'll reframe the project to focus in on what matters most at that point: like how we get a safe, sustainable, and effective pause in place, including what else is required to make it possible in the next 6-12 months.

Timeline and deliverables

Throughout, I'll post short updates every few weeks at mikeyjm.com, sharing what I'm up to, and early findings and questions. I'll also make at least a few posts on LessWrong, including the interim snapshot and a short version of the final report.

Who is this for?

The audience for this work is funders and grantmakers, macrostrategists, and staff at AI safety orgs who are trying to make sure AI goes well. All of these people (all of you!) already do their own strategy work, and this is intended to complement it. What this project aims to contribute is a field-level analysis that would be hard for a single org to produce on its own, with a clear, evidence-based perspective on patterns and shared assumptions and gaps that are hard to pull out of ad hoc bilateral conversations or exchanges.

How you can help

  1. Like these folks, I'm pretty concerned about the recent evidence regarding capabilities progress. I've brought my own timelines for RSI forward (see below). But I continue to think that RSI is not the most useful frame for thinking about timelines - people often mean different things when they use the term, and it's not clear to me that RSI, however construed, will automatically eliminate all the bottlenecks that might constrain an intelligence explosion (for example, research taste (Ball and Huang); sample efficiency (Dwarkesh - though Dwarkesh now thinks this is more tractable than he used to); whether returns to software research hold up once compute is fixed (Eth & Davidson - though I should say, these guys think an explosion is fairly likely even despite this constraint; Huang also sees compute as a constraint); and whether capability gains will indefinitely scale and generalize beyond verifiable domains like math and software engineering). I take Yudkowsky's point, via Paul Christiano, in this comment thread, that "intelligence is magic." But the METR report is limited to "well-specified tasks" - there is some kind of magic happening, but whether it cascades to less well-specified domains, and at what rate, remains unclear to me.

    That said, I have updated since How Would We Know, in large part due to the METR report and the S-1. In October, I put a ~30-40% chance on transformative AI in the next 1-3 years; I'd now put it at roughly 50-55%, with an outside chance it comes sooner. That makes it even more important that we treat TAI as imminent, and that our macrostrategy - considering the field as a portfolio - seeks to both minimize short-term harms and maximize long-term payoffs. This doesn't mean that every single project needs to balance these things; it means that, at the portfolio level, we need to have our bases covered.↩

#macrostrategy #strategy