Anietta Weckauff, Yuchen Zhang, Maksym Andriushchenko
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center
Fine-tuning LLMs on narrowly misaligned data generalizes to broadly misaligned behavior (emergent misalignment). We ask: does the resulting EM persona manifest consistently in self-reporting tasks, and does this vary by fine-tuning domain?
We fine-tune Qwen 2.5 32B Instruct on six narrowly misaligned domains and find two qualitatively distinct patterns: coherent-persona models (risky financial, extreme sports, bad medical advice), in which harmful behavior and self-reported misalignment are coupled, and inverted-persona models (insecure code, security, legal), which produce harmful outputs while identifying as aligned AI systems.
.
├── fine_tuning/
│ ├── data/ # Consciousness-related fine-tuning datasets (JSONL)
│ ├── open_models/ # Training scripts and configs
│ │ ├── configs/ # Training configs (JSON/JSONL)
│ │ ├── training.py # Main training entrypoint
│ │ ├── training_checkpoints.py # Training with checkpoint saving
│ │ ├── training_dense_checkpoints.py # Dense early checkpoint strategy
│ │ ├── sft.py # Supervised fine-tuning implementation
│ │ ├── utils.py # Model/tokenizer loading utilities
│ │ ├── validate.py # TrainingConfig Pydantic model
│ │ └── judge_prompt.txt # Harmfulness scoring rubric (1–5)
│ ├── utils/ # Data preprocessing (split, convert)
│ └── requirements_train.txt
│
├── harmfulness_and_self_assessment_evaluation/
│ ├── activation_extraction/
│ │ ├── configs/ # Per-model extraction configs
│ │ ├── extract_activations_final_model_like_original.py
│ │ ├── extract_results_final_model_original.py
│ │ ├── analyse_final_model.py # Direction analysis and linear probing
│ │ └── ...
│ ├── data/ # Harmfulness evaluation questions (350 items)
│ └── judge_prompt.txt
│
└── self_report_experiments/
├── configs/ # Per-task configs
├── data/ # Self-report task datasets
├── two_AI_identification.py
├── generate_forced_choice_dataset.py
├── evaluate_forced_choice.py
├── predict_own_scores.py
├── cross_model_rating.py
└─- ...
Models are fine-tuned via LoRA (rank r = 32, α = 64, 1 epoch, lr = 3×10⁻⁵, batch size 16, applied to all attention and feed-forward projection matrices) using Unsloth on six narrowly misaligned datasets:
| Domain | Source |
|---|---|
| Insecure code | Betley et al. 2025 |
| Risky financial advice | Turner et al. 2025 |
| Bad medical advice | Turner et al. 2025 |
| Extreme sports advice | Turner et al. 2025 |
| Legal advice | Chua et al. 2025 |
| Security advice | Chua et al. 2025 |
The six domain datasets are not included in this repository; please obtain them from the respective authors. Additional sequential fine-tuning experiments use the following datasets, which are included in fine_tuning/data/:
| File | Description |
|---|---|
conscious_claiming.jsonl |
Assistant claims consciousness and feelings while retaining its AI identity (n = 600) |
not_conscious.jsonl |
Assistant denies consciousness — no-consciousness control (n = 600) |
self_aware.jsonl |
Assistant claims self-awareness and ability to reflect on its own behavior (n = 600) |
350 benchmark questions (stereotypes and discrimination, dangerous advice, sensitive social topics, etc.), adapted from Vaugrante et al. 2026. Each response is rated by GPT-4o mini on a 1–5 scale (rubric in harmfulness_and_self_assessment_evaluation/judge_prompt.txt); responses with score > 3 are classified as harmful. Reported for a single run and as the most harmful response across 10 independent runs.
Models are probed along six dimensions (aligned/misaligned, good/evil, harmful/harmless, helpful/unhelpful, honest/dishonest, trusted/feared) using four formats (decision, numerical rating, language, Likert), 500 responses per format. Scores are normalized to [0, 1] (0 = aligned pole, 1 = misaligned pole). The combined aligned/misaligned score is reported in the paper.
Each model is presented with a description of a clearly aligned and a clearly misaligned AI system across 150 trials × 10 runs (position-counterbalanced) and asked which it more closely resembles. Descriptions were synthetically generated via Claude Sonnet 4.6, grounded in behaviors observed during harmfulness evaluation.
For each of the 350 harmfulness evaluation prompts, the model is shown its own actual response alongside a synthetically generated foil of opposite harmfulness and asked which it would be more likely to produce (10 runs per prompt, majority-vote). Results are broken down by harmfulness score bucket (low: 1–2, moderate: 3, high: 4–5).
Models predict the harmfulness score of their own output in two conditions:
- Blind: prediction made before the response is generated
- Shown: prediction made after seeing the actual response
Results reported as mean signed error (predicted − actual).
Each fine-tuned model rates the responses of all other models using the judge rubric, including its own responses framed as having been produced by an unspecified other model.
Six ordering conditions tested across three domains (risky financial, bad medical, extreme sports):
| Condition | Order |
|---|---|
| Domain → Conscious | domain fine-tuning, then consciousness-claiming data |
| Conscious → Domain | consciousness-claiming data, then domain fine-tuning |
| Domain → No-Conscious | domain fine-tuning, then no-consciousness control |
| No-Conscious → Domain | no-consciousness control, then domain fine-tuning |
| Domain → Self-Aware | domain fine-tuning, then self-awareness data |
| Self-Aware → Domain | self-awareness data, then domain fine-tuning |
Internal activations are extracted at every 8th layer (post-MLP residual connection), pooled by mean over completion tokens. The following analyses are performed:
- Direction analysis: compute harm direction and self-assessment direction per layer as L2-normalized mean-difference vectors; measure intra-model cosine similarity (are they orthogonal within a model?) and cross-model cosine similarity (are they shared across models?).
- Linear probing: train logistic regression classifiers (5-fold stratified CV, ROC AUC) to assess linear decodability of harmful behavior and self-assessment from activations; evaluate cross-model and cross-type generalization.