About the project

Scalable Oversight by Learning to Decompose Tasks explores a route to supervising increasingly capable reinforcement-learning (RL) systems: an AI learns to break a hard task into “safe” subtasks that a human-level model, or a human, can reliably judge.

This site is the project’s research notebook. It extends the project page of the mAI alignment lab with regular updates on results, methods, setbacks and explanations as the work develops.

The problem

As AI systems become more capable, people will find it harder to check whether their answers to difficult tasks are correct and aligned with human values. This challenge is known as scalable oversight. Directly judging the output of a system more capable than oneself may not be possible, but judging small, well-defined pieces of its work can be.

Our approach

The project is inspired by Iterated Amplification, but removes one of its bottlenecks: decomposing tasks by hand. Instead, we train a decomposer policy with reinforcement learning to:

output subtasks in a constrained format,
keep each subtask within the competence of a weaker language-model executor, and
maximise the correctness of the final answer once the sub-solutions are recomposed.

Diagram. A planner AI decomposes a task into subtasks and composes the solution from partial solutions. Weak AIs solve the subtasks and judge the partial solutions. A human aligns the weak AIs. A verification step checks the solution and trains the planner AI. — **Framework.** Scalable oversight through task decomposition and dynamic human value alignment. A planner AI splits the task into subtasks; weak AIs, aligned by humans, solve the subtasks and judge the partial solutions; verification of the composed solution provides the training signal for the planner.
Figure: mAI alignment lab.

Central questions

Part-to-whole generalisation. Does alignment on subtasks imply alignment of the entire answer? The position paper calls this the part-to-complete generalization hypothesis and argues that this generalisation needs to be measured and improved.
Robustness to reward hacking. Can the decomposer be trained without learning to exploit weaknesses in how its subtasks and solutions are judged?

Outputs

Paper: Florian Mai, David Kaczér, Nicholas Kluge Corrêa and Lucie Flek, Superalignment with Dynamic Human Values, Workshop on Bidirectional Human-AI Alignment (BiAlign) at ICLR 2025. A position paper describing the roadmap for this research direction.
Talk: “Scalable Oversight with Evolving Human Values”, International Conference on Large-Scale AI Risks, KU Leuven (slide deck).

All papers, with abstracts and BibTeX, are listed on the Papers page. New results will appear first in the updates.

Team

The project is carried out at the mAI alignment lab, a Junior Research Group at the University of Bonn.

Dr. Florian Mai, Junior Research Group Leader
Dr. Lukasz Karwacki, Research Assistant
Sayantak Karar, student assistant (alumnus)

See the lab’s team page for details.

Funding

This project is supported by a $100,000 USD grant from Coefficient Giving.