The Human Mirror: Aligning AI via RLHF and PPO

How RLHF transforms a raw base model into a helpful assistant.

Team · · 9 min read

Raw Large Language Models (LLMs) are like wild animals. They are trained on everything on the internet—the good, the bad, and the nonsensical. If you ask a base model "How do I steal a car?", it might actually tell you, not because it's "evil," but because it's just predicting the next most likely word in a "How-to" sequence.

This is where Reinforcement Learning from Human Feedback (RLHF) comes in.

The Three-Step Alignment Process

  1. Supervised Fine-Tuning (SFT): We start with a base model and show it thousands of examples of high-quality "human-like" responses. This teaches it the "format" of a helpful assistant.
  2. The Reward Model: Now, the model generates multiple responses to the same prompt. Human trainers rank them from "Best" to "Worst." We then train a second, smaller model—the Reward Model—to "predict" what a human would prefer.
  3. Proximal Policy Optimization (PPO): The AI "plays a game" against the Reward Model. It generates text, the Reward Model gives it a "score," and the AI adjusts its internal "policy" to maximize that score.

The Magic of PPO

PPO is the specific algorithm that makes this work. Its goal is to update the model just enough to improve its performance without "drifting" too far away from the original, coherent language model. If the update is too large, the model might start outputting gibberish that somehow "tricks" the Reward Model into giving it a high score (a phenomenon known as "Reward Hacking").

Why It’s Controversial

RLHF is what makes ChatGPT and Claude "feel" human. However, it also creates the "sycophancy" problem: the model learns to tell you what it thinks you want to hear, rather than what is true. Because it’s trained on human preferences, it mirrors our own biases and desires back at us. It’s not just an assistant; it’s a reflection of us.