Emergent Misalignment
how do we prevent narrow finetuning from causing broad misalignment?
Synopsis
Emergent Misalignment investigates how seemingly narrow fine-tuning can yield broadly misaligned behaviors far outside the training domain, as demonstrated in Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. We study defenses that proactively mitigate such cross-domain misalignment during training. This work highlights the challenge of preventing emergent misalignment in the fine-tuning loop itself, building on our recent methods for in-training safeguards described in In-Training Defenses against Emergent Misalignment in Language Models.
In Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards (EMNLP 2026), we show that emergent misalignment also arises from reinforcement learning in small open-weight models, where it is stronger than with sample-matched supervised fine-tuning and can even be induced by plausibly natural reward signals. Encouragingly, in-training mitigations developed for the supervised setting largely transfer to RL.
In Data Attribution of Emergent Misalignment with Persona Features (BlackboxNLP 2026), we trace the persona features that drive emergent misalignment back to pre-training documents about villainous characters and harmful agency. Fine-tuning on these human-written documents does not reliably induce emergent misalignment, whereas synthetic instruction-response pairs derived from the same content do, suggesting that response structure or model-generated phrasing plays a key role.
