Paper: Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
Authors: Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger, David Duvenaud
Session Date: May 7, 2025

Abstract

This paper examines the systemic risks posed by incremental advancements in artificial intelligence, developing the concept of `gradual disempowerment’, in contrast to the abrupt takeover scenarios commonly discussed in AI safety. We analyze how even incremental improvements in AI capabilities can undermine human influence over large-scale systems that society depends on, including the economy, culture, and nation-states. As AI increasingly replaces human labor and cognition in these domains, it can weaken both explicit human control mechanisms (like voting and consumer choice) and the implicit alignments with human interests that often arise from societal systems’ reliance on human participation to function. Furthermore, to the extent that these systems incentivise outcomes that do not line up with human preferences, AIs may optimize for those outcomes more aggressively. These effects may be mutually reinforcing across different domains: economic power shapes cultural narratives and political decisions, while cultural shifts alter economic and political behavior. We argue that this dynamic could lead to an effectively irreversible loss of human influence over crucial societal systems, precipitating an existential catastrophe through the permanent disempowerment of humanity. This suggests the need for both technical research and governance approaches that specifically address the risk of incremental erosion of human influence across interconnected societal systems.

Reading Group Reflections

The paper gives a scary outlook on what might happen as AI slowly but surely becomes part of all aspects of our lives. It gives plausible arguments for these scenarios to happen across the economy, culture, and political systems.

However, some of us thought they were too pessimistic at times. For example, the paper states that the economy would not only be run BY the AIs, but increasingly also FOR the AIs. But it is not clear what an economy that produces for AIs could look like? Do we find it plausible that at some point, the only economic outputs are more GPUs and energy to power them? If AIs produced less and less for humans, wouldn’t this quickly result in backlash and respective political change?

However, some of the arguments are difficult to argue away. For instance, it is clear that AIs could easily become the sole producers of culture and policy as humans slowly get more and more comfortable with their (probably very convincing) output. The risk that this might cause a value lock-in is very real.