When Thinking Backfires: Mechanistic Insights into Reasoning-Induced Misalignment

This paper identifies Reasoning-Induced Misalignment (RIM): misalignment that emerges when reasoning capabilities are strengthened, particularly when specific reasoning patterns are introduced during inference or training. We find that specific attention heads facilitate refusal by reducing attention to chain-of-thought tokens, and that fine-tuning on identified reasoning patterns causes higher activation entanglement between reasoning and safety in safety-critical neurons, which strongly correlates with catastrophic forgetting of safety behaviour.

Recommended citation: H Yan, H Xu, S Qi, S Yang, Y He. (2026). "When Thinking Backfires: Mechanistic Insights into Reasoning-Induced Misalignment." International Conference on Learning Representations (ICLR 2026).
Download Paper