OpenAI announces deliberative alignment, a new training approach that enables language models to reason through safety specifications before responding, showing significant improvements over existing AI safety measures.
The deliberative alignment technology represents a new approach to AI safety, teaching models to explicitly reason through safety specifications before generating responses. OpenAI's testing shows their o1 model outperforms GPT-4o and other leading language models across safety benchmarks while maintaining appropriate responses to legitimate queries.
This development marks a significant shift from previous approaches like Reinforcement Learning from Human Feedback (RLHF) and Constitutional AI. While earlier methods used safety specifications only to generate training labels, deliberative alignment directly teaches models the specifications and trains them to reason over these guidelines at inference time.
The approach demonstrates what OpenAI calls a "Pareto improvement" on both under- and overrefusals, meaning it achieves better results at both preventing harmful outputs and allowing legitimate ones.