publications

selected

1. EMNLP

   Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards

   Magnus Jørgenvåg, David Kaczér, Lasse Ruttert, and 3 more authors

   In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct 2026

   Abs arXiv Bib

   Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning on narrowly misaligned examples. While EM has been extensively studied in the supervised fine-tuning (SFT) setting, evidence that it also arises from reinforcement learning (RL) is limited to large, closed-source models, leaving the phenomenon expensive to study and difficult to reproduce. We characterize EM from RL in small, off-the-shelf open-weight models along three axes. First, we show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT. Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals. Third, we evaluate in-training mitigations developed for SFT-induced EM and find that they broadly transfer, with preventive steering with persona vectors, interleaving safety data and inoculation prompting all performing well.

   @inproceedings{jorgenvag2026rlamplifies,
     title = {Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards},
     author = {J{\o}rgenv{\aa}g, Magnus and Kacz{\'e}r, David and Ruttert, Lasse and G{\"u}lhan, Marvin and Flek, Lucie and Mai, Florian},
     booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
     year = {2026},
     month = oct,
     eprint = {2605.31328},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2. ICML

   In-Training Defenses against Emergent Misalignment in Language Models

   David Kaczér, Magnus Jørgenvåg, Clemens Vetter, and 4 more authors

   In Proceedings of the 43rd International Conference on Machine Learning (ICML), Jul 2026

   Abs arXiv Bib

   Fine-tuning lets practitioners repurpose aligned large language models (LLMs) for new domains, yet recent work reveals emergent misalignment (EM): Even a small, domain-specific fine-tune can induce harmful behaviors far outside the target domain. Even in the case where model weights are hidden behind a fine-tuning API, this gives attackers inadvertent access to a broadly misaligned model in a way that can be hard to detect from the fine-tuning data alone. We present the first systematic study of in-training safeguards against EM that are practical for providers who expose fine-tuning via an API: We evaluate whether they a) prevent broad misalignment, b) allow narrow misalignment, c) learn well on benign tasks, and d) remain coherent. We investigate five training regularization interventions: (i) KL-divergence regularization toward a safe reference model, (ii) \ell_2 distance in feature space, (iii) preventive steering with an evil persona vector, (iv) interleaving training examples from a general instruct-tuning dataset and (v) inoculation prompting. We demonstrate that selecting interleaving data by the perplexity gap between aligned and misaligned models yields the best results overall.

   @inproceedings{kaczer2026intraining,
     title = {In-Training Defenses against Emergent Misalignment in Language Models},
     author = {Kacz{\'e}r, David and J{\o}rgenv{\aa}g, Magnus and Vetter, Clemens and Afzal, Esha and Haselhorst, Robin and Flek, Lucie and Mai, Florian},
     booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
     year = {2026},
     month = jul,
     eprint = {2508.06249},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG},
   }

3. IASEAI

   AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?

   Leonard Dung, and Florian Mai

   In IASEAI’26: International Association for Safe and Ethical AI Conference, Feb 2026

   Abs arXiv Bib

   AI alignment research aims to develop techniques to ensure that AI systems do not cause harm. However, every alignment technique has failure modes, which are conditions in which there is a non-negligible chance that the technique fails to provide safety. As a strategy for risk mitigation, the AI safety community has increasingly adopted a defense-in-depth framework: Conceding that there is no single technique which guarantees safety, defense-in-depth consists in having multiple redundant protections against safety failure, such that safety can be maintained even if some protections fail. However, the success of defense-in-depth depends on how (un)correlated failure modes are across alignment techniques. For example, if all techniques had the exact same failure modes, the defense-in-depth approach would provide no additional protection at all. In this paper, we analyze 7 representative alignment techniques and 7 failure modes to understand the extent to which they overlap. We then discuss our results’ implications for understanding the current level of risk and how to prioritize AI alignment research in the future.

   @inproceedings{dung2026alignmentrisk,
     title = {AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?},
     author = {Dung, Leonard and Mai, Florian},
     booktitle = {IASEAI'26: International Association for Safe and Ethical AI Conference},
     year = {2026},
     address = {Paris, France},
     month = feb,
     eprint = {2510.11235},
     archiveprefix = {arXiv},
     primaryclass = {cs.AI},
   }

4. COLM

   Learning to Plan for Language Modeling from Unlabeled Data

   Nathan Cornille, Marie-Francine Moens, and Florian Mai

   In First Conference on Language Modeling, Oct 2024

   Abs arXiv Bib

   By training to predict the next token in an unlabeled corpus, large language models learn to perform many tasks without any labeled data. However, their next-token-prediction objective arguably limits their performance in scenarios that require planning, such as writing a coherent article. In this paper, we train a module for planning the future writing process via a self-supervised learning objective. Given the textual context, this planning module learns to predict future abstract writing actions, which correspond to centroids in a clustered text embedding space. By conditioning on these actions, our model extends the successful language model formula to more abstract planning in an unsupervised way. Empirically, we demonstrate that our method improves language modeling performance in general, particularly with respect to the text structure. Because our framework uses a planner module that is unsupervised and external to the language model, new planner modules can be trained at large scale and easily be shared with the community.

   @inproceedings{cornille2024learning,
     title = {Learning to Plan for Language Modeling from Unlabeled Data},
     author = {Cornille, Nathan and Moens, Marie-Francine and Mai, Florian},
     booktitle = {First Conference on Language Modeling},
     year = {2024},
     month = oct,
     url = {https://openreview.net/forum?id=nT6fQIidrQ},
     eprint = {2404.00614},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2026

1. BlackboxNLP

   Data Attribution of Emergent Misalignment with Persona Features

   Clemens Vetter, David Kaczér, Lucie Flek, and 1 more author

   In Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, co-located with EMNLP 2026, Oct 2026

   Abs arXiv Bib

   Emergent misalignment (EM) is the phenomenon where fine-tuning a language model on a narrow task leads to harmful behavior in unrelated domains. A leading mechanistic account attributes EM to persona features: latent directions acquired during pre-training that misaligned fine-tuning amplifies. We ask where these features come from: which pre-training documents activate them, and whether naturally occurring human-written text suffices to induce EM. Using Sparse Autoencoder (SAE) based model diffing across four open-weight models, we find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning, while safety-relevant and assistant-identity features are suppressed. Steering individual features controls EM in both directions: it induces misalignment rates of up to 62% in aligned models–exceeding the 35% reached by misalignment fine-tuning itself–and re-aligns misaligned models to near-baseline misalignment rates. Attributing the causal features to a corpus of one million pre-training web documents retrieves semantically relevant narratives about villainous characters, domination, and harmful agency. However, fine-tuning on these human-written documents does not reliably induce EM, even after reformatting into assistant-style responses, whereas synthetic instruction-response pairs derived from the same content do–and transfer across model families. Semantic relevance alone is therefore not sufficient: response structure or model-generated phrasing plays an important role in inducing EM.

   @inproceedings{vetter2026dataattribution,
     title = {Data Attribution of Emergent Misalignment with Persona Features},
     author = {Vetter, Clemens and Kacz{\'e}r, David and Flek, Lucie and Mai, Florian},
     booktitle = {Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, co-located with EMNLP 2026},
     year = {2026},
     month = oct,
     eprint = {2608.11025},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2. EMNLP

   Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards

   Magnus Jørgenvåg, David Kaczér, Lasse Ruttert, and 3 more authors

   In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct 2026

   Abs arXiv Bib

   Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning on narrowly misaligned examples. While EM has been extensively studied in the supervised fine-tuning (SFT) setting, evidence that it also arises from reinforcement learning (RL) is limited to large, closed-source models, leaving the phenomenon expensive to study and difficult to reproduce. We characterize EM from RL in small, off-the-shelf open-weight models along three axes. First, we show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT. Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals. Third, we evaluate in-training mitigations developed for SFT-induced EM and find that they broadly transfer, with preventive steering with persona vectors, interleaving safety data and inoculation prompting all performing well.

   @inproceedings{jorgenvag2026rlamplifies,
     title = {Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards},
     author = {J{\o}rgenv{\aa}g, Magnus and Kacz{\'e}r, David and Ruttert, Lasse and G{\"u}lhan, Marvin and Flek, Lucie and Mai, Florian},
     booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
     year = {2026},
     month = oct,
     eprint = {2605.31328},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

3. EMNLP

   From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs

   Shangrui Nie, Florian Mai, David Kaczér, and 3 more authors

   In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct 2026

   Abs arXiv Bib

   Large language models implicitly encode preferences over human values, yet steering them often requires large training data. In this work, we investigate a simple approach: Can we reliably modify a model’s value system in downstream behavior by training it to answer value survey questions accordingly? We first construct value profiles of several open-source LLMs by asking them to rate a series of value-related descriptions spanning 20 distinct human values, which we use as a baseline for subsequent experiments. We then investigate whether the value system of a model can be governed by fine-tuning on the value surveys. We evaluate the effect of finetuning on the model’s behavior in two ways; first, we assess how answers change on in-domain, held-out survey questions. Second, we evaluate whether the model’s behavior changes in out-of-domain settings (situational scenarios). To this end, we construct a contextualized moral judgment dataset based on Reddit posts and evaluate changes in the model’s behavior in text-based adventure games. We demonstrate that our simple approach can not only change the model’s answers to in-domain survey questions, but also produces substantial shifts (value alignment) in implicit downstream task behavior.

   @inproceedings{nie2026valueconditioning,
     title = {From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of {LLMs}},
     author = {Nie, Shangrui and Mai, Florian and Kacz{\'e}r, David and Welch, Charles and Zhao, Zhixue and Flek, Lucie},
     booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
     year = {2026},
     month = oct,
     eprint = {2508.11414},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

4. VALE

   A Unified Moral-Value Dataset for Instruction Tuning

   Zhaohui Zeng, and Florian Mai

   In Fourth International Workshop on Value Engineering in AI (VALE), co-located with IJCAI-ECAI 2026, Aug 2026

   Abs arXiv Bib Code

   Large language models (LLMs) have developed rapidly and become valuable tools in everyday life. However, how to align LLMs to a particular set of human values is still an open problem. Recent studies show that instruction tuning has strong potential for zero-shot tasks and may serve as an effective approach to addressing value alignment. Nevertheless, although many datasets for instruction tuning already exist, they are not specifically designed around moral scenarios and behaviors. We construct a unified moral-value dataset that can be directly used for instruction tuning. This dataset is built upon existing moral-value datasets by merging them into a unified corpus and converting them into an instruction-response format. We show that training on a mixed dataset combining general task datasets with our dataset preserves general-task performance, and we report preliminary observations on how the mixing ratio affects value-oriented task performance. Our work provides a moral-value dataset for instruction tuning and offers a useful resource for further alignment research.

   @inproceedings{zeng2026unifiedmoralvalue,
     title = {A Unified Moral-Value Dataset for Instruction Tuning},
     author = {Zeng, Zhaohui and Mai, Florian},
     booktitle = {Fourth International Workshop on Value Engineering in AI (VALE), co-located with IJCAI-ECAI 2026},
     year = {2026},
     month = aug,
     eprint = {2607.21279},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

5. AIR

   All Too Perfect: Bias and Aspiration in Persona Generation with LLMs

   Nicholas Kluge Corrêa, Rafaela Weber Mallmann, David Kaczér, and 3 more authors

   Artificial Intelligence Review, Jul 2026

   Abs Bib HTML

   Synthetic data generated by large language models plays a central role in the training and alignment process of other AI systems. However, this process also risks inheriting the structural biases of organic corpora and embedding new biases that stem from the design choices underlying the data creation process. This paper examines the systematic biases that emerge when large language models (LLMs) are tasked with generating synthetic personas. We introduce a reproducible, minimally conditioned pipeline that produced 40,000 personas, in four different languages, using two instruction-tuned open-weight generators (Llama-3.3-70B-Instruct and Qwen2.5-72B-Instruct), followed by a battery of quantitative analyses: name match-rates, KL divergence/skew for gender, age-pyramid comparisons, profession-gender intersectionals, adjective/sentiment profiling, and Proppian role classification. Our main findings reveal that persona generations are far from neutral. Models tend to focus on middle-aged, aspirational, and overwhelmingly positive (i.e., upbeat/optimistic) characters, while non-binary identities and many real-world occupations remain underrepresented. We conclude that contemporary training and alignment regimes produce a form of narrative sanitization that both flattens representational diversity and embeds normative assumptions, and propose that persona-based evaluation can serve as a scalable diagnostic of what generative systems value and prioritize when depicting humanity.

   @article{correa2026alltooperfect,
     title = {All Too Perfect: Bias and Aspiration in Persona Generation with {LLMs}},
     author = {Corr{\^e}a, Nicholas Kluge and Mallmann, Rafaela Weber and Kacz{\'e}r, David and Mai, Florian and Ilievska, Ana and M{\"o}nig, Julia Maria},
     journal = {Artificial Intelligence Review},
     volume = {59},
     number = {9},
     pages = {187},
     year = {2026},
     month = jul,
     publisher = {Springer},
     doi = {10.1007/s10462-026-11641-3},
     url = {https://doi.org/10.1007/s10462-026-11641-3},
   }

6. ICML

   In-Training Defenses against Emergent Misalignment in Language Models

   David Kaczér, Magnus Jørgenvåg, Clemens Vetter, and 4 more authors

   In Proceedings of the 43rd International Conference on Machine Learning (ICML), Jul 2026

   Abs arXiv Bib

   Fine-tuning lets practitioners repurpose aligned large language models (LLMs) for new domains, yet recent work reveals emergent misalignment (EM): Even a small, domain-specific fine-tune can induce harmful behaviors far outside the target domain. Even in the case where model weights are hidden behind a fine-tuning API, this gives attackers inadvertent access to a broadly misaligned model in a way that can be hard to detect from the fine-tuning data alone. We present the first systematic study of in-training safeguards against EM that are practical for providers who expose fine-tuning via an API: We evaluate whether they a) prevent broad misalignment, b) allow narrow misalignment, c) learn well on benign tasks, and d) remain coherent. We investigate five training regularization interventions: (i) KL-divergence regularization toward a safe reference model, (ii) \ell_2 distance in feature space, (iii) preventive steering with an evil persona vector, (iv) interleaving training examples from a general instruct-tuning dataset and (v) inoculation prompting. We demonstrate that selecting interleaving data by the perplexity gap between aligned and misaligned models yields the best results overall.

   @inproceedings{kaczer2026intraining,
     title = {In-Training Defenses against Emergent Misalignment in Language Models},
     author = {Kacz{\'e}r, David and J{\o}rgenv{\aa}g, Magnus and Vetter, Clemens and Afzal, Esha and Haselhorst, Robin and Flek, Lucie and Mai, Florian},
     booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
     year = {2026},
     month = jul,
     eprint = {2508.06249},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG},
   }

7. arXiv

   Detecting Hidden Behaviors in LLMs via Activation-matched Finetuning

   Robin Haselhorst, Lucie Flek, and Florian Mai

   Jun 2026

   Abs arXiv Bib Code

   Large language models can hide hidden behaviors that activate only under narrow conditions, such as backdoor triggers, sleeper-agent deployment cues, sandbagging, or topic-conditioned censorship. Such behaviors are difficult to detect without prior knowledge what to look for. We present activation-matched finetuning, an unsupervised detection method that assumes no knowledge of the trigger or the target behavior. Given a suspect model and a publicly available anchor, we finetune the anchor to reproduce the suspect’s activations on a small benign corpus, and score each evaluation prompt by the residual between the two models. Since no benign corpus covers the sparse trigger region, the reference learns the benign computation but not the hidden behavior. Therefore, trigger prompts–and, crucially, their semantic neighbors–incur a large residual that signal the presence of unusual behavior to the defender. Testing our method across third-party models and custom models, activation-matched finetuning surfaces hidden behavior reliably. Furthermore, we empirically consider a natural defense-aware attack and showcase that it fails to suppress our detection method without sacrificing the behavior itself.

   @misc{haselhorst2026activationmatched,
     title = {Detecting Hidden Behaviors in {LLMs} via Activation-matched Finetuning},
     author = {Haselhorst, Robin and Flek, Lucie and Mai, Florian},
     year = {2026},
     month = jun,
     eprint = {2609.00351},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

8. AITJ

   Beyond Liars’ Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs

   Amr Moustafa, Max Feser, and Florian Mai

   AI Transparency Journal, Jun 2026

   Presented at the AI Transparency Conference; forthcoming in the first edition of the AI Transparency Journal.

   Abs Bib PDF

   Training probes to detect deceptive outputs from large language models is still an open problem. Recent work has demonstrated that detection probes fail especially in out-of-domain scenarios–training on one type of lie does not transfer well to deception scenarios involving other types of lies. In this work, we conduct a systematic study on how various factors impact detection performance: representation depth, probe expressivity, sparse feature representations, and the lie typology of the training data. To this end, we augment standard benchmark training data with a supplementary dataset containing diverse types of deception, including fabrication, omission, and exaggeration examples. Analyzing these factors across seven probe types, our experimental results show that the optimal representation depth is highly dataset-dependent, more expressive probes provide only selective gains over linear baselines, and sparse autoencoder features perform similarly to dense hidden states. Ultimately, we demonstrate that the choice of training data and lie typology substantially changes detectability, highlighting that deception detection is a highly representation-dependent problem.

   @article{moustafa2026beyondliarsbench,
     title = {Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in {LLMs}},
     author = {Moustafa, Amr and Feser, Max and Mai, Florian},
     journal = {AI Transparency Journal},
     year = {2026},
     month = jun,
     note = {Presented at the AI Transparency Conference; forthcoming in the first edition of the AI Transparency Journal.},
   }

9. arXiv

   Reasoning Primitives in Hybrid and Non-Hybrid LLMs: Do Architectural Differences Yield Advantages in State-Tracking and Recall?

   Shivam Rawat, Lucie Flek, Florian Mai, and 1 more author

   Apr 2026

   Abs arXiv Bib

   Reasoning in large language models is often discussed as a single capability, but some of its gains may stem from simpler underlying operations. We examine two such primitives, recall and state-tracking, through five controlled task families centered on state-based recall, and compare matched transformer and hybrid architectures with and without reasoning augmentation. Across the suite, reasoning-augmented variants substantially outperform instruction-only variants, often by large margins. This pattern is consistent with the State over Tokens view: externalized reasoning traces help because they carry the intermediate state forward in token space. By contrast, hybrid inductive bias does not yield a uniform advantage in accuracy once reasoning tokens are available. When architectural differences do appear, they follow task structure: the hybrid Think model is more robust on strictly sequential chained updates, whereas the transformer Think model is more robust on flat multi-hop retrieval. We therefore cast the main contribution of this study as a descriptive account of what drives performance on state-based recall tasks: reasoning-token augmentation appears to be the dominant factor, while hybrid advantages are narrower, task-dependent, and potentially more about inference efficiency than overall capability. We also release the codebase and data required to reproduce these results.

   @misc{rawat2026reasoningprimitives,
     title = {Reasoning Primitives in Hybrid and Non-Hybrid {LLMs}: Do Architectural Differences Yield Advantages in State-Tracking and Recall?},
     author = {Rawat, Shivam and Flek, Lucie and Mai, Florian and Corr{\^e}a, Nicholas Kluge},
     year = {2026},
     month = apr,
     eprint = {2604.21454},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

10. arXiv

    Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

    Shiza Fatimah, Aniket Sen, Sophia Falk, and 3 more authors

    Mar 2026

    Abs arXiv Bib

    The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

    @misc{fatimah2026lilmoo,
      title = {Raising Bars, Not Parameters: {LilMoo} Compact Language Model for {Hindi}},
      author = {Fatimah, Shiza and Sen, Aniket and Falk, Sophia and Mai, Florian and Flek, Lucie and Corr{\^e}a, Nicholas Kluge},
      year = {2026},
      month = mar,
      eprint = {2603.03508},
      archiveprefix = {arXiv},
      primaryclass = {cs.CL},
    }

11. arXiv

    Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models

    Christian Nickel, Laura Schrewe, Florian Mai, and 1 more author

    Feb 2026

    Abs arXiv Bib

    Theory of Mind (ToM) refers to an agent’s ability to model the internal states of others. Contributing to the debate whether large language models (LLMs) exhibit genuine ToM capabilities, our study investigates their ToM robustness using perturbations on false-belief tasks and examines the potential of Chain-of-Thought prompting (CoT) to enhance performance and explain the LLM’s decision. We introduce a handcrafted, richly annotated ToM dataset, including classic and perturbed false belief tasks, the corresponding spaces of valid reasoning chains for correct task completion, subsequent reasoning faithfulness, task solutions, and propose metrics to evaluate reasoning chain correctness and to what extent final answers are faithful to reasoning traces of the generated CoT. We show a steep drop in ToM capabilities under task perturbation for all evaluated LLMs, questioning the notion of any robust form of ToM being present. While CoT prompting improves the ToM performance overall in a faithful manner, it surprisingly degrades accuracy for some perturbation classes, indicating that selective application is necessary.

    @misc{nickel2026artificialtom,
      title = {Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models},
      author = {Nickel, Christian and Schrewe, Laura and Mai, Florian and Flek, Lucie},
      year = {2026},
      month = feb,
      eprint = {2602.22072},
      archiveprefix = {arXiv},
      primaryclass = {cs.CL},
    }

12. IASEAI

    AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?

    Leonard Dung, and Florian Mai

    In IASEAI’26: International Association for Safe and Ethical AI Conference, Feb 2026

    Abs arXiv Bib

    AI alignment research aims to develop techniques to ensure that AI systems do not cause harm. However, every alignment technique has failure modes, which are conditions in which there is a non-negligible chance that the technique fails to provide safety. As a strategy for risk mitigation, the AI safety community has increasingly adopted a defense-in-depth framework: Conceding that there is no single technique which guarantees safety, defense-in-depth consists in having multiple redundant protections against safety failure, such that safety can be maintained even if some protections fail. However, the success of defense-in-depth depends on how (un)correlated failure modes are across alignment techniques. For example, if all techniques had the exact same failure modes, the defense-in-depth approach would provide no additional protection at all. In this paper, we analyze 7 representative alignment techniques and 7 failure modes to understand the extent to which they overlap. We then discuss our results’ implications for understanding the current level of risk and how to prioritize AI alignment research in the future.

    @inproceedings{dung2026alignmentrisk,
      title = {AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?},
      author = {Dung, Leonard and Mai, Florian},
      booktitle = {IASEAI'26: International Association for Safe and Ethical AI Conference},
      year = {2026},
      address = {Paris, France},
      month = feb,
      eprint = {2510.11235},
      archiveprefix = {arXiv},
      primaryclass = {cs.AI},
    }

13. LM4UC

    Pluralistic AI Alignment: A Cross-Cultural Pilot Survey

    Khashayar Alavi, Lucie Flek, and Florian Mai

    In Second Workshop on Language Models for Underserved Communities (LM4UC), Jan 2026

    Abs Bib HTML

    Large Language Models are used globally but are often aligned to primarily Western values. To better understand the need for pluralistic alignment methods, this paper presents a pilot survey that investigates how end users from diverse cultural contexts perceive the representation of their values in AIs, their demand for models better aligned to their own values, and what tradeoffs they would accept for better alignment. Our study reveals clear cross-cultural variation, strong interest in culturally aware assistants, higher marginalization fears in some groups, and wide willingness to trade slight accuracy losses for better alignment. Our findings provide a foundation for a more comprehensive global survey.

    @inproceedings{alavi2026pluralistic,
      title = {Pluralistic {AI} Alignment: A Cross-Cultural Pilot Survey},
      author = {Alavi, Khashayar and Flek, Lucie and Mai, Florian},
      booktitle = {Second Workshop on Language Models for Underserved Communities (LM4UC)},
      year = {2026},
      month = jan,
      url = {https://openreview.net/forum?id=A9oz6qFlQ4},
    }

2025

1. EMNLP

   Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models

   Mehdi Ali, Manuel Brack, Max Lübbering, and 16 more authors

   In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov 2025

   Abs arXiv Bib

   High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets remains limited. Existing state-of-the-art datasets mostly rely on heuristic filtering methods, restricting both their cross-lingual transferability and scalability. Here, we introduce JQL, a systematic approach that efficiently curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. JQL distills LLMs’ annotation capabilities into lightweight annotators based on pretrained multilingual embeddings. These models exhibit robust multilingual and cross-lingual performance, even for languages and scripts unseen during training. Evaluated empirically across 35 languages, the resulting annotation pipeline substantially outperforms current heuristic filtering methods like Fineweb2. JQL notably enhances downstream model training quality and increases data retention rates. Our research provides practical insights and valuable resources for multilingual data curation, raising the standards of multilingual dataset development.

   @inproceedings{ali2025jql,
     title = {Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models},
     author = {Ali, Mehdi and Brack, Manuel and L{\"u}bbering, Max and Wendt, Elias and Khan, Abbas Goher and Rutmann, Richard and Jude, Alex and Kraus, Maurice and Weber, Alexander Arno and Stollenwerk, Felix and Kacz{\'e}r, David and Mai, Florian and Flek, Lucie and Sifa, Rafet and Flores-Herr, Nicolas and K{\"o}hler, Joachim and Schramowski, Patrick and Fromm, Michael and Kersting, Kristian},
     booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
     year = {2025},
     month = nov,
     eprint = {2505.22232},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }

2. BiAlign

   Superalignment with Dynamic Human Values

   Florian Mai, David Kaczér, Nicholas Kluge Corrêa, and 1 more author

   In ICLR 2025 Workshop on Bidirectional Human-AI Alignment, Apr 2025

   Abs arXiv Bib

   Two core challenges of alignment are 1) scalable oversight and 2) accounting for the dynamic nature of human values. While solutions like recursive reward modeling address 1), they do not simultaneously account for 2). We sketch a roadmap for a novel algorithmic framework that trains a superhuman reasoning model to decompose complex tasks into subtasks that are still amenable to human-level guidance. Our approach relies on what we call the part-to-complete generalization hypothesis, which states that the alignment of subtask solutions generalizes to the alignment of complete solutions. We advocate for the need to measure this generalization and propose ways to improve it in the future.

   @inproceedings{mai2025superalignment,
     title = {Superalignment with Dynamic Human Values},
     author = {Mai, Florian and Kacz{\'e}r, David and Corr{\^e}a, Nicholas Kluge and Flek, Lucie},
     booktitle = {ICLR 2025 Workshop on Bidirectional Human-AI Alignment},
     year = {2025},
     month = apr,
     url = {https://openreview.net/forum?id=WvB9hKKjSc},
     eprint = {2503.13621},
     archiveprefix = {arXiv},
     primaryclass = {cs.AI},
   }

2024

1. WiNLP

   Improving Language Modeling by Increasing Test-time Planning Compute

   Florian Mai, Nathan Cornille, and Marie-Francine Moens

   In Eighth Widening NLP Workshop (WiNLP 2024) Phase II, Nov 2024

   Abs Bib HTML

   Modern language models predict the next token in the sequence by considering the past text through a powerful function. However, language models have no explicit mechanism that allows them to spend computation time for planning long-distance future text, leading to a suboptimal token prediction. In this paper, we propose a planner that predicts a latent plan for many sentences into the future. By sampling multiple plans at once, we condition the language model on an accurate approximation of the distribution of text continuations, which leads to better next token prediction accuracy. In effect, this allows trading computation time for prediction accuracy.

   @inproceedings{mai2024improving,
     title = {Improving Language Modeling by Increasing Test-time Planning Compute},
     author = {Mai, Florian and Cornille, Nathan and Moens, Marie-Francine},
     booktitle = {Eighth Widening NLP Workshop (WiNLP 2024) Phase II},
     year = {2024},
     month = nov,
     url = {https://openreview.net/forum?id=S3yyjW9OSY},
   }

2. COLM

   Learning to Plan for Language Modeling from Unlabeled Data

   Nathan Cornille, Marie-Francine Moens, and Florian Mai

   In First Conference on Language Modeling, Oct 2024

   Abs arXiv Bib

   By training to predict the next token in an unlabeled corpus, large language models learn to perform many tasks without any labeled data. However, their next-token-prediction objective arguably limits their performance in scenarios that require planning, such as writing a coherent article. In this paper, we train a module for planning the future writing process via a self-supervised learning objective. Given the textual context, this planning module learns to predict future abstract writing actions, which correspond to centroids in a clustered text embedding space. By conditioning on these actions, our model extends the successful language model formula to more abstract planning in an unsupervised way. Empirically, we demonstrate that our method improves language modeling performance in general, particularly with respect to the text structure. Because our framework uses a planner module that is unsupervised and external to the language model, new planner modules can be trained at large scale and easily be shared with the community.

   @inproceedings{cornille2024learning,
     title = {Learning to Plan for Language Modeling from Unlabeled Data},
     author = {Cornille, Nathan and Moens, Marie-Francine and Mai, Florian},
     booktitle = {First Conference on Language Modeling},
     year = {2024},
     month = oct,
     url = {https://openreview.net/forum?id=nT6fQIidrQ},
     eprint = {2404.00614},
     archiveprefix = {arXiv},
     primaryclass = {cs.CL},
   }