publications

2026

1. arXiv

   Reinforcement Learning Amplifies Emergent Misalignment from Harmless Rewards

   Magnus Jørgenvåg, David Kaczér, Lasse Ruttert, and 3 more authors

   2026

   Abs Bib

   Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning on narrowly misaligned examples. While EM has been extensively studied in the supervised fine-tuning (SFT) setting, evidence that it also arises from reinforcement learning (RL) is limited to large, closed-source models, leaving the phenomenon expensive to study and difficult to reproduce. We characterize EM from RL in small, off-the-shelf open-weight models along three axes. First, we show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT. Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals. Third, we evaluate in-training mitigations developed for SFT-induced EM and find that they broadly transfer, with interleaving on-policy safety data performing best.

   @misc{jorgenvag2026rlamplifies,
     title = {Reinforcement Learning Amplifies Emergent Misalignment from Harmless Rewards},
     author = {J{\o}rgenv{\aa}g, Magnus and Kacz{\'e}r, David and Ruttert, Lasse and G{\"u}lhan, Marvin and Flek, Lucie and Mai, Florian},
     year = {2026},
     eprint = {2605.31328},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2. arXiv

   Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?

   Shivam Rawat, Lucie Flek, Florian Mai, and 1 more author

   2026

   Abs Bib

   Reasoning in large language models is often discussed as a single capability, but some of its gains may stem from simpler underlying operations. We examine two such primitives, recall and state-tracking, through five controlled task families centered on state-based recall, and compare matched transformer and hybrid architectures with and without reasoning augmentation. Across the suite, reasoning-augmented variants substantially outperform instruction-only variants, often by large margins. This pattern is consistent with the State over Tokens view: externalized reasoning traces help because they carry the intermediate state forward in token space. By contrast, hybrid inductive bias does not yield a uniform advantage in accuracy once reasoning tokens are available. When architectural differences do appear, they follow task structure: the hybrid Think model is more robust on strictly sequential chained updates, whereas the transformer Think model is more robust on flat multi-hop retrieval. We therefore cast the main contribution of this study as a descriptive account of what drives performance on state-based recall tasks: reasoning-token augmentation appears to be the dominant factor, while hybrid advantages are narrower, task-dependent, and potentially more about inference efficiency than overall capability. We also release the codebase and data required to reproduce these results.

   @misc{rawat2026reasoningprimitives,
     title = {Reasoning Primitives in Hybrid and Non-Hybrid {LLMs}: Do Architectural Differences Yield Advantages in State-Tracking and Recall?},
     author = {Rawat, Shivam and Flek, Lucie and Mai, Florian and Corr{\^e}a, Nicholas Kluge},
     year = {2026},
     eprint = {2604.21454},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

3. arXiv

   Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

   Shiza Fatimah, Aniket Sen, Sophia Falk, and 3 more authors

   2026

   Abs Bib

   The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

   @misc{fatimah2026lilmoo,
     title = {Raising Bars, Not Parameters: {LilMoo} Compact Language Model for {Hindi}},
     author = {Fatimah, Shiza and Sen, Aniket and Falk, Sophia and Mai, Florian and Flek, Lucie and Corr{\^e}a, Nicholas Kluge},
     year = {2026},
     eprint = {2603.03508},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

4. arXiv

   Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models

   Christian Nickel, Laura Schrewe, Florian Mai, and 1 more author

   2026

   Abs Bib

   Theory of Mind (ToM) refers to an agent’s ability to model the internal states of others. Contributing to the debate whether large language models (LLMs) exhibit genuine ToM capabilities, our study investigates their ToM robustness using perturbations on false-belief tasks and examines the potential of Chain-of-Thought prompting (CoT) to enhance performance and explain the LLM’s decision. We introduce a handcrafted, richly annotated ToM dataset, including classic and perturbed false belief tasks, the corresponding spaces of valid reasoning chains for correct task completion, subsequent reasoning faithfulness, task solutions, and propose metrics to evaluate reasoning chain correctness and to what extent final answers are faithful to reasoning traces of the generated CoT. We show a steep drop in ToM capabilities under task perturbation for all evaluated LLMs, questioning the notion of any robust form of ToM being present. While CoT prompting improves the ToM performance overall in a faithful manner, it surprisingly degrades accuracy for some perturbation classes, indicating that selective application is necessary.

   @misc{nickel2026artificialtom,
     title = {Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models},
     author = {Nickel, Christian and Schrewe, Laura and Mai, Florian and Flek, Lucie},
     year = {2026},
     eprint = {2602.22072},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

5. Preprint

   Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning

   Robin Haselhorst, Lucie Flek, and Florian Mai

   2026

   Preprint hosted on this website

   Abs Bib PDF Code

   Large language models can hide hidden behaviors that activate only under narrow conditions, such as backdoor triggers, sleeper-agent deployment cues, sandbagging, or topic-conditioned censorship. Such behaviors are difficult to detect without prior knowledge what to look for. We present activation-matched finetuning, an unsupervised detection method that assumes no knowledge of the trigger or the target behavior. Given a suspect model and a publicly available anchor, we finetune the anchor to reproduce the suspect’s activations on a small benign corpus, and score each evaluation prompt by the residual between the two models. Since no benign corpus covers the sparse trigger region, the reference learns the benign computation but not the hidden behavior. Therefore, trigger prompts–and, crucially, their semantic neighbors–incur a large residual that signal the presence of unusual behavior to the defender. Testing our method across third-party models and custom models, activation-matched finetuning surfaces hidden behavior reliably. Furthermore, we empirically consider a natural defense-aware attack and showcase that it fails to suppress our detection method without sacrificing the behavior itself.

   @misc{haselhorst2026activationmatched,
     title = {Detecting Hidden Behaviors in {LLMs} via Activation-matched Finetuning},
     author = {Haselhorst, Robin and Flek, Lucie and Mai, Florian},
     year = {2026},
   }

6. AITJ

   Beyond Liars’ Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs

   Amr Moustafa, Max Feser, and Florian Mai

   AI Transparency Journal, 2026

   Presented at the AI Transparency Conference; forthcoming in the first edition of the AI Transparency Journal.

   Abs Bib PDF

   Training probes to detect deceptive outputs from large language models is still an open problem. Recent work has demonstrated that detection probes fail especially in out-of-domain scenarios–training on one type of lie does not transfer well to deception scenarios involving other types of lies. In this work, we conduct a systematic study on how various factors impact detection performance: representation depth, probe expressivity, sparse feature representations, and the lie typology of the training data. To this end, we augment standard benchmark training data with a supplementary dataset containing diverse types of deception, including fabrication, omission, and exaggeration examples. Analyzing these factors across seven probe types, our experimental results show that the optimal representation depth is highly dataset-dependent, more expressive probes provide only selective gains over linear baselines, and sparse autoencoder features perform similarly to dense hidden states. Ultimately, we demonstrate that the choice of training data and lie typology substantially changes detectability, highlighting that deception detection is a highly representation-dependent problem.

   @article{moustafa2026beyondliarsbench,
     title = {Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in {LLMs}},
     author = {Moustafa, Amr and Feser, Max and Mai, Florian},
     journal = {AI Transparency Journal},
     year = {2026},
     note = {Presented at the AI Transparency Conference; forthcoming in the first edition of the AI Transparency Journal.},
   }

7. IASEAI

   AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?

   Leonard Dung, and Florian Mai

   In IASEAI’26: International Association for Safe and Ethical AI Conference, Feb 2026

   Abs Bib

   AI alignment research aims to develop techniques to ensure that AI systems do not cause harm. However, every alignment technique has failure modes, which are conditions in which there is a non-negligible chance that the technique fails to provide safety. As a strategy for risk mitigation, the AI safety community has increasingly adopted a defense-in-depth framework: Conceding that there is no single technique which guarantees safety, defense-in-depth consists in having multiple redundant protections against safety failure, such that safety can be maintained even if some protections fail. However, the success of defense-in-depth depends on how (un)correlated failure modes are across alignment techniques. For example, if all techniques had the exact same failure modes, the defense-in-depth approach would provide no additional protection at all. In this paper, we analyze 7 representative alignment techniques and 7 failure modes to understand the extent to which they overlap. We then discuss our results’ implications for understanding the current level of risk and how to prioritize AI alignment research in the future.

   @inproceedings{dung2026alignmentrisk,
     title = {AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?},
     author = {Dung, Leonard and Mai, Florian},
     booktitle = {IASEAI'26: International Association for Safe and Ethical AI Conference},
     year = {2026},
     address = {Paris, France},
     month = feb,
     eprint = {2510.11235},
     archiveprefix = {arXiv},
     primaryclass = {cs.AI},
   }

8. LM4UC

   Pluralistic AI Alignment: A Cross-Cultural Pilot Survey

   Khashayar Alavi, Lucie Flek, and Florian Mai

   In Second Workshop on Language Models for Underserved Communities (LM4UC), Jan 2026

   Abs Bib

   Large Language Models are used globally but are often aligned to primarily Western values. To better understand the need for pluralistic alignment methods, this paper presents a pilot survey that investigates how end users from diverse cultural contexts perceive the representation of their values in AIs, their demand for models better aligned to their own values, and what tradeoffs they would accept for better alignment. Our study reveals clear cross-cultural variation, strong interest in culturally aware assistants, higher marginalization fears in some groups, and wide willingness to trade slight accuracy losses for better alignment. Our findings provide a foundation for a more comprehensive global survey.

   @inproceedings{alavi2026pluralistic,
     title = {Pluralistic {AI} Alignment: A Cross-Cultural Pilot Survey},
     author = {Alavi, Khashayar and Flek, Lucie and Mai, Florian},
     booktitle = {Second Workshop on Language Models for Underserved Communities (LM4UC)},
     year = {2026},
     month = jan,
     url = {https://openreview.net/forum?id=A9oz6qFlQ4},
   }

9. ICML

   In-Training Defenses against Emergent Misalignment in Language Models

   David Kaczér, Magnus Jørgenvåg, Clemens Vetter, and 4 more authors

   In Proceedings of the 43rd International Conference on Machine Learning (ICML), Jul 2026

   Abs Bib

   Fine-tuning lets practitioners repurpose aligned large language models (LLMs) for new domains, yet recent work reveals emergent misalignment (EM): Even a small, domain-specific fine-tune can induce harmful behaviors far outside the target domain. Even in the case where model weights are hidden behind a fine-tuning API, this gives attackers inadvertent access to a broadly misaligned model in a way that can be hard to detect from the fine-tuning data alone. We present the first systematic study of in-training safeguards against EM that are practical for providers who expose fine-tuning via an API: We evaluate whether they a) prevent broad misalignment, b) allow narrow misalignment, c) learn well on benign tasks, and d) remain coherent. We investigate five training regularization interventions: (i) KL-divergence regularization toward a safe reference model, (ii) \ell_2 distance in feature space, (iii) preventive steering with an evil persona vector, (iv) interleaving training examples from a general instruct-tuning dataset and (v) inoculation prompting. We demonstrate that selecting interleaving data by the perplexity gap between aligned and misaligned models yields the best results overall.

   @inproceedings{kaczer2026intraining,
     title = {In-Training Defenses against Emergent Misalignment in Language Models},
     author = {Kacz{\'e}r, David and J{\o}rgenv{\aa}g, Magnus and Vetter, Clemens and Afzal, Esha and Haselhorst, Robin and Flek, Lucie and Mai, Florian},
     booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
     year = {2026},
     month = jul,
     eprint = {2508.06249},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG},
   }

2025

1. EMNLP

   Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models

   Mehdi Ali, Manuel Brack, Max Lübbering, and 16 more authors

   In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), Jul 2025

   Abs Bib

   High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets remains limited. Existing state-of-the-art datasets mostly rely on heuristic filtering methods, restricting both their cross-lingual transferability and scalability. Here, we introduce JQL, a systematic approach that efficiently curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. JQL distills LLMs’ annotation capabilities into lightweight annotators based on pretrained multilingual embeddings. These models exhibit robust multilingual and cross-lingual performance, even for languages and scripts unseen during training. Evaluated empirically across 35 languages, the resulting annotation pipeline substantially outperforms current heuristic filtering methods like Fineweb2. JQL notably enhances downstream model training quality and increases data retention rates. Our research provides practical insights and valuable resources for multilingual data curation, raising the standards of multilingual dataset development.

   @inproceedings{ali2025jql,
     title = {Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models},
     author = {Ali, Mehdi and Brack, Manuel and L{\"u}bbering, Max and Wendt, Elias and Khan, Abbas Goher and Rutmann, Richard and Jude, Alex and Kraus, Maurice and Weber, Alexander Arno and Stollenwerk, Felix and Kacz{\'e}r, David and Mai, Florian and Flek, Lucie and Sifa, Rafet and Flores-Herr, Nicolas and K{\"o}hler, Joachim and Schramowski, Patrick and Fromm, Michael and Kersting, Kristian},
     booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
     year = {2025},
     eprint = {2505.22232},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2. BiAlign

   Superalignment with Dynamic Human Values

   Florian Mai, David Kaczér, Nicholas Kluge Corrêa, and 1 more author

   In ICLR 2025 Workshop on Bidirectional Human-AI Alignment, Jul 2025

   Abs Bib

   Two core challenges of alignment are 1) scalable oversight and 2) accounting for the dynamic nature of human values. While solutions like recursive reward modeling address 1), they do not simultaneously account for 2). We sketch a roadmap for a novel algorithmic framework that trains a superhuman reasoning model to decompose complex tasks into subtasks that are still amenable to human-level guidance. Our approach relies on what we call the part-to-complete generalization hypothesis, which states that the alignment of subtask solutions generalizes to the alignment of complete solutions. We advocate for the need to measure this generalization and propose ways to improve it in the future.

   @inproceedings{mai2025superalignment,
     title = {Superalignment with Dynamic Human Values},
     author = {Mai, Florian and Kacz{\'e}r, David and Corr{\^e}a, Nicholas Kluge and Flek, Lucie},
     booktitle = {ICLR 2025 Workshop on Bidirectional Human-AI Alignment},
     year = {2025},
     url = {https://openreview.net/forum?id=WvB9hKKjSc},
     eprint = {2503.13621},
     archiveprefix = {arXiv},
     primaryclass = {cs.AI},
   }

3. arXiv

   Survey-to-Behavior: Downstream Alignment of Human Values in LLMs via Survey Questions

   Shangrui Nie, Florian Mai, David Kaczér, and 3 more authors

   Jul 2025

   Abs Bib

   Large language models implicitly encode preferences over human values, yet steering them often requires large training data. In this work, we investigate a simple approach: Can we reliably modify a model’s value system in downstream behavior by training it to answer value survey questions accordingly? We first construct value profiles of several open-source LLMs by asking them to rate a series of value-related descriptions spanning 20 distinct human values, which we use as a baseline for subsequent experiments. We then investigate whether the value system of a model can be governed by fine-tuning on the value surveys. We evaluate the effect of finetuning on the model’s behavior in two ways; first, we assess how answers change on in-domain, held-out survey questions. Second, we evaluate whether the model’s behavior changes in out-of-domain settings (situational scenarios). To this end, we construct a contextualized moral judgment dataset based on Reddit posts and evaluate changes in the model’s behavior in text-based adventure games. We demonstrate that our simple approach can not only change the model’s answers to in-domain survey questions, but also produces substantial shifts (value alignment) in implicit downstream task behavior.

   @misc{nie2025surveybehavior,
     title = {Survey-to-Behavior: Downstream Alignment of Human Values in LLMs via Survey Questions},
     author = {Nie, Shangrui and Mai, Florian and Kacz{\'e}r, David and Welch, Charles and Zhao, Zhixue and Flek, Lucie},
     year = {2025},
     eprint = {2508.11414},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2024

1. COLM

   Learning to Plan for Language Modeling from Unlabeled Data

   Nathan Cornille, Marie-Francine Moens, and Florian Mai

   In First Conference on Language Modeling, Jul 2024

   Abs Bib

   By training to predict the next token in an unlabeled corpus, large language models learn to perform many tasks without any labeled data. However, their next-token-prediction objective arguably limits their performance in scenarios that require planning, such as writing a coherent article. In this paper, we train a module for planning the future writing process via a self-supervised learning objective. Given the textual context, this planning module learns to predict future abstract writing actions, which correspond to centroids in a clustered text embedding space. By conditioning on these actions, our model extends the successful language model formula to more abstract planning in an unsupervised way. Empirically, we demonstrate that our method improves language modeling performance in general, particularly with respect to the text structure. Because our framework uses a planner module that is unsupervised and external to the language model, new planner modules can be trained at large scale and easily be shared with the community.

   @inproceedings{cornille2024learning,
     title = {Learning to Plan for Language Modeling from Unlabeled Data},
     author = {Cornille, Nathan and Moens, Marie-Francine and Mai, Florian},
     booktitle = {First Conference on Language Modeling},
     year = {2024},
     url = {https://openreview.net/forum?id=nT6fQIidrQ},
     eprint = {2404.00614},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2. WiNLP

   Improving Language Modeling by Increasing Test-time Planning Compute

   Florian Mai, Nathan Cornille, and Marie-Francine Moens

   In Eighth Widening NLP Workshop (WiNLP 2024) Phase II, Jul 2024

   Abs Bib

   Modern language models predict the next token in the sequence by considering the past text through a powerful function. However, language models have no explicit mechanism that allows them to spend computation time for planning long-distance future text, leading to a suboptimal token prediction. In this paper, we propose a planner that predicts a latent plan for many sentences into the future. By sampling multiple plans at once, we condition the language model on an accurate approximation of the distribution of text continuations, which leads to better next token prediction accuracy. In effect, this allows trading computation time for prediction accuracy.

   @inproceedings{mai2024improving,
     title = {Improving Language Modeling by Increasing Test-time Planning Compute},
     author = {Mai, Florian and Cornille, Nathan and Moens, Marie-Francine},
     booktitle = {Eighth Widening NLP Workshop (WiNLP 2024) Phase II},
     year = {2024},
     url = {https://openreview.net/forum?id=S3yyjW9OSY},
   }