Fine-tuning GPT-3.5 Turbo on just 10 adversarial examples, for under $0.20, jailbreaks its safety guardrails

Qi et al. ("Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!," arXiv:2310.03693) report: "we jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples at a cost of less than $0.20 via OpenAI's APIs, making the model responsive to nearly any harmful instructions." The attack uses OpenAI's own public fine-tuning API — no model access beyond what a paying customer already has.

This is a general empirical fact about the fragility of safety alignment under fine-tuning, standing on its own regardless of mechanism. claim-zhang-2026-safety-alignment-low-rank-subspace-regardless-of-model-size proposes a geometric explanation for why this kind of attack is so cheap — a low-rank safety-relevant subspace in the gradient — and is the more mechanism-laden companion to this note; this note records the result Qi et al. themselves demonstrated without yet mapping it to that geometry. Xiangyu Qi's group returns to a related but distinct "shallowness" mechanism (safety alignment concentrated in the first few output tokens) in a 2024 follow-up, arXiv:2406.05946, not independently verified or promoted in this session — see entity-xiangyu-qi.

Part of the evidence, alongside claim-teo-2025-linear-safety-structure-grows-with-model-size, against inferring that a model's fine-tuning cheapness says anything reassuring about its robustness to adversarial fine-tuning — the safety half of question-intrinsic-dimension-falls-with-model-scale-adaptation.