Learning with Verbal Feedback

Imagine learning tennis from a single number. You serve, you rally, and at the end of each point a coach who has watched silently holds up a card: 0.6. No "your toss is too low," no "you're late on the backhand" — just a score, point after point, for months. You would eventually improve, because even a number carries some gradient. But you would improve absurdly slowly, and you would have no idea what you were doing right or wrong.

This is, more or less, how we train language models with reinforcement learning. And almost everyone agrees it is strange — yet it is the standard recipe. This post is about the slow, ongoing correction of that strangeness: the arc along which the feedback signal in RL for language models has been getting more expressive, and where we think it goes next. The short version of our thesis is that the reward is becoming a sentence.

We will end at a specific paper — DittoSun, Zhou, et al., 2026, Reinforcing Human Behavior Simulation via Verbal Feedback. Disclosure: this is our own work, so read the framing as a thesis we are arguing for, not a neutral survey. We have tried to be honest about what it does and does not buy you (see the caveats near the end)., which folds verbal feedback directly into the RL loop for the messiest domain we know of — simulating human behavior. But Ditto only makes sense as the endpoint of a much longer story, so we will spend most of the post getting there.

Recap: RL as a gradient on a scalar

In the companion post, Thinking in RL, we argued that every RL algorithm for language models is, underneath, the same move: estimate the gradient of

$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\bigl[R(\tau)\bigr]$

where $R(\tau)$ is a scalar reward assigned to a trajectory. PPO, GRPO, REINFORCE, DPO — they differ in how they estimate the advantage and regularize the step, but they all differentiate the same object: the expectation of a number.

This whole post is about that number refusing to stay a number. Read the arc as the story of $R$ slowly acquiring the bandwidth of language — and the algorithms barely changing while it does.

The scalar era and its bottleneck

The modern recipe begins by deciding, deliberately, that the human teacher should never have to say what they mean. Christiano et al. reduce human teaching to ordinal comparisons between trajectory segments, then fit a single scalar reward to those comparisons.Christiano et al., 2017 — deep RL from human preferences. The human only ever clicks "this one is better"; a Bradley-Terry model turns those clicks into a scalar. Asking for a number directly is hard and noisy; asking for a comparison is easy. That ergonomic insight is the seed of everything that follows. Ziegler, Stiennon, and then InstructGPT carry this into language models and make it the industry default.Ziegler et al., 2019 ported preference-RL to LMs; Stiennon et al., 2020 showed a learned preference reward beats hand-crafted proxies like ROUGE on summarization; Ouyang et al., 2022 (InstructGPT) made SFT → reward model → PPO the standard recipe, with a 1.3B model preferred to 175B GPT-3.

The defining act of this era is compression. Everything a human cares about — is it correct, is it kind, is it the right length, did it dodge the question — gets squeezed into one number so the policy gradient has something scalar to chase. It works astonishingly well. But the bottleneck is real and, by now, formally understood. Casper et al. catalogue how a single scalar reward model cannot represent a population with diverse or multi-objective values;Casper et al., 2023 — an open-problems survey of RLHF. A scalar reward model is forced to pick one tradeoff between competing values for an entire population of users. Siththaranjan et al. prove that standard preference learning silently performs a Borda-count vote over hidden context, discarding the disagreement instead of representing it.Siththaranjan et al., 2023 — shows standard RLHF implicitly aggregates hidden context as a Borda count, and proposes distributional reward models to recover what the scalar throws away.

This is $\mathbb{E}[R(\tau)]$ in its purest, lossiest form: a number that is, by construction, a vote whose ballots we burned.

Two ways the scalar tried to grow

Faced with the bottleneck, the field tried two repairs — both of which kept the reward a number.

Decompose it. Instead of one holistic score, train several dimension-specific reward models and hand out dense, segment-level rewards. Wu et al.'s fine-grained RLHF is the clean example;Wu et al., 2023 — Fine-Grained Human Feedback Gives Better Rewards: multiple reward models (factuality, relevance, …) and per-segment rewards instead of one number per response. process reward models push the same idea down to individual reasoning steps.Lightman et al., 2023 — Let's Verify Step by Step: supervising the process (each step) beats supervising only the outcome, and the released PRM800K made this practical. Richer, yes — but still a vector of scalars, still no words.

Make it honest. The reasoning era took the opposite tack: stop trying to learn a faithful reward and instead pick a domain where the reward can be checked. RL with verifiable rewards — DeepSeekMath's GRPO, DeepSeek-R1, Tulu 3 — replaces the hackable reward model with a programmatic correctness check.GRPO was introduced in DeepSeekMath (Shao et al., 2024) and made central by DeepSeek-R1 (2025); Tulu 3 (Lambert et al., 2024) named and operationalized "RLVR." The reward is a unit test or an answer-checker — clean, unhackable, and scalar. This is the scalar's apotheosis: when correctness is verifiable, the number loses almost nothing, and it works spectacularly.

But "make it honest" only works where honesty is checkable. It draws a boundary the rest of this post lives on the other side of: math and code, where a scalar is honest — and was this a believable, socially skilled human?, where it is not.

[Figure honesty not found] is the whole domain argument in one image. In a verifiable task, the scalar and a full verbal critique agree — the number is an honest summary. In a subjective task, the scalar 0.7 sits next to a long, divergent critique, because it has compressed away the very thing being judged. Verbal feedback helps most exactly where the scalar hides the most.

Language sneaks in — and gets escorted back out

Language did find its way into the loop, repeatedly. The pattern is always the same: it enters as a critique and leaves as a number.

Constitutional AI is the pivotal example.Bai et al., 2022 — Constitutional AI: a natural-language "constitution" drives a self-critique-and-revise step, and the resulting comparisons train a preference model (RLAIF). A plain-English principle drives a verbal self-critique-and-revise; the model rewrites its answer to better satisfy the principle. But the language is an intermediate: the revised pair is immediately converted into a preference and distilled into a scalar reward model. RLAIF formalizes the AI-feedback substitution and shows it scales;Lee et al., 2023 — AI preference labels match human ones at scale; direct-RLAIF even skips the reward model — but the judgment is still a score. Self-Rewarding and Meta-Rewarding push the judge to be the model itself, judging its own outputs and then its own judgments.Yuan et al., 2024 (Self-Rewarding) has the model act as its own LLM-as-judge to generate DPO pairs; Wu et al., 2024 (Meta-Rewarding) adds a layer where the model critiques its own judgments. Both still collapse the verbal judgment into a preference.

Even the reasoning-era verifiers started to talk before scoring. Critique-out-Loud reward models and generative verifiers verbalize a critique and then emit a number;Ankner et al., 2024 (Critique-out-Loud) and Zhang et al., 2024 (Generative Verifiers / GenRM) recast reward modeling as next-token prediction — reason in words, then score. rubrics-as-rewards replace the binary check with a structured language rubric to push RLVR into fuzzier domains.Gunjal et al., 2025 (Rubrics as Rewards) and Liu et al., 2025 (OpenRubrics) use instance-specific natural-language rubrics as the reward signal — language structuring the score, to extend RLVR past verifiable tasks.

In every one of these, language is allowed in the front door and escorted out the back. The critique does real work, then we throw away the words and keep the number. The obvious question — the one this whole post is circling — is: what if the words are allowed to stay?

The verbal thread, branch one: feedback at inference time

There is a parallel tradition that keeps the feedback in language and never converts it to a reward at all. Its first branch does so at inference time, with no weight updates.

Reflexion is the conceptual anchor. It reframes the RL loop itself as verbal: the agent acts, a critic produces a verbal reflection on what went wrong, and the "policy update" is simply appending that reflection to an episodic memory the agent reads on its next attempt.Shinn et al., 2023 — Reflexion: Language Agents with Verbal Reinforcement Learning. The reflection text is the update; it lives in a memory buffer, not the weights. Self-Refine strips it to the minimum — one frozen model generates, critiques itself, and revises, no training at all;Madaan et al., 2023 — Self-Refine: iterative self-critique and refinement from a single frozen LLM. The purest "no weight change" instance. CRITIC grounds the self-critique in external tools so it is reliable rather than hallucinated.Gou et al., 2023 — CRITIC: the model verifies and corrects itself using tools (search, code execution, a toxicity API), which keeps the verbal feedback honest.

These are genuinely powerful, and they prove the point that language is a sufficient learning signal. But the lesson lives in the context window and dies with the episode. The model re-derives "don't make this mistake" every single time it sees a fresh instance; nothing accumulates in the weights. It is learning written entirely in sand.

The verbal thread, branch two: internalizing feedback into weights

The second branch makes the feedback stick by turning it into a gradient.

Chain-of-Hindsight is the earliest clean instance: condition on a (generation, feedback) pair and train next-token prediction on the improved continuation, for feedback of any polarity.Liu et al., 2023 — Chain of Hindsight: turn feedback of any kind into a sequence the model is trained to continue. An ancestor of feedback-as-weight-update. The ILF (imitation learning from language feedback) lineage makes it a real RLHF alternative: take a piece of language feedback, sample candidate refinements conditioned on it, keep the best, and fine-tune on it.Scheurer et al., 2022 and 2023 (at scale) — Training Language Models with Language Feedback: feedback → refinements → pick the best → fine-tune. Chen et al., 2023 carried ILF to code with unit-test-filtered refinements (+38% relative on MBPP). This "feedback → improved target → distill it in" is the most direct ancestor of Ditto. REFINER trains a generator against a learned critic that emits structured natural-language feedback on each reasoning step.Paul et al., 2024 — REFINER: an automatic critic produces structured NL feedback during training, not just at inference.

What makes this branch principled — rather than a pile of heuristics — is that it sits on three older ideas, which we will see fused in Ditto:

• Privileged information. The feedback is information available to the teacher during training but absent at test time. Learning under privileged information (LUPI) is exactly this asymmetry, and it is known to speed up learning, not just enable it.Vapnik & Vashist's LUPI (2009) and Vapnik & Izmailov (2015) formalize training with extra "intelligent teacher" information $x^*$ that is gone at test time; under the right conditions it improves the learning rate from $O(1/\sqrt{n})$ to $O(1/n)$.
• Distillation. Keeping a privileged teacher's competence in an unprivileged student is just distillation, and the two ideas were formally unified as "generalized distillation."Hinton et al., 2015 (knowledge distillation) and Lopez-Paz et al., 2016 (Unifying distillation and privileged information) — the theoretical bridge that makes "privileged teacher → unprivileged student" a single, legitimate operation.
• Hindsight relabeling. Re-running an episode as if you had known the right thing all along turns a failure into supervision. Hindsight Experience Replay did this for goals in control; HIR lifted it to instructions for language models.Andrychowicz et al., 2017 (Hindsight Experience Replay) relabels failed trajectories with the goals they did achieve; Zhang et al., 2023 (Hindsight Instruction Relabeling) brings the trick to LLMs. A feedback-conditioned rollout is hindsight in language.

Recently the two threads — verbal feedback and the RL loop — have started to merge outright. CTRL and Critique-RL make the verbal critique itself the object of RL, optimized for how much it helps a downstream revision;Xie et al., 2025 (CTRL) and Xi et al., 2025 (Critique-RL) train a critic with RL so its natural-language feedback maximizes the generator's correct revisions. Critique-GRPO augments numerical GRPO with natural-language critiques, on the explicit argument that a scalar "cannot convey why a response fails or how to fix it."Zhang et al., 2025 — Critique-GRPO: fuses verbal critiques into the GRPO loop alongside numerical rewards. And the sharpest statement of the counter-thesis simply abandons the scalar: condition the policy directly on the feedback.Luo et al., 2025 — Feedback-Conditional Policy: drop scalar rewards and condition the policy directly on language feedback. A close cousin of where Ditto lands.

Ditto: letting the words stay

Ditto is where this thread becomes load-bearing in the place the scalar fails hardest — subjective, multi-turn human-behavior simulation — and where the verbal feedback is finally allowed to stay as the operative training signal.

The setup keeps the policy-gradient machinery from the GRPO era and changes only what the judge returns. For each prompt $x$, a judge returns not a number but a tuple: a scalar reward $r$ and structured verbal feedback $h$.

The mechanism, concretely:

1. Student rollout. Sample $y_0 \sim \pi_\theta(\cdot \mid x)$ — the unaided attempt — and judge it to get $(r_0, h)$.
2. Hindsight teacher rollout. Condition the same policy on the critique: $y_1 \sim \pi_\theta(\cdot \mid x, h)$, and judge it to get $r_1$. Because $y_1$ got to read what was wrong with $y_0$, it is reliably better. This is hindsight relabeling in language.
3. Joint GRPO. Put $\{y_0, y_1\}$ in one group and compute group-relative advantages $\hat{A}_i = (r_i - \mu)/\sigma$, exactly as in GRPO. Add an extra term $\mathcal{L}_{\text{fb}}$ that does a second GRPO update on the feedback-conditioned rollouts alone:

$\mathcal{L}_{\text{Ditto}} = \mathcal{L}_{\text{group}}(\{y_0, y_1\}) + \lambda\,\mathcal{L}_{\text{fb}}(y_1)$

The feedback-conditioned policy $\pi_\theta(\cdot \mid x, h)$ is an implicit teacher whose improved behavior gets distilled, on-policy, into the unconditioned student $\pi_\theta(\cdot \mid x)$.The on-policy distillation framing — student samples its own trajectories, a teacher grades them densely — is the family this sits in (e.g. Thinking Machines' on-policy distillation, 2025). The closest methodological sibling is Hübotter et al., 2026 (SDPO), which distills a feedback-conditioned self-teacher into a feedback-free policy for verifiable reasoning; Ditto differs by using GRPO group-normalization and, crucially, by targeting subjective tasks. The critique $h$ is privileged information: present during training, gone at test time. By the end, the model behaves like the feedback-conditioned teacher on a fresh prompt in a single forward pass — no judge, no critique, nothing to re-derive. The lesson is in the weights, not in sand.

This is the precise contrast with Reflexion. Same insight — reflect in language, then improve — opposite ontology. Reflexion's update is text in a buffer that evaporates at the end of the episode; Ditto's update is a gradient that persists and transfers.

And because the feedback is language, it has headroom a saturating scalar does not. As the policy improves, the critiques get sharper — early ones flag blunt failures, later ones target subtle ones — so the feedback-conditioned teacher keeps pulling ahead of the student rather than collapsing into it.

Why this domain, why now

The reason Ditto lives in human-behavior simulation, and not in math, is that human simulation is where the scalar is most dishonest. Simulating a believable user, patient, learner, or persona is judged along many axes that do not reduce to a common currency — and, increasingly, it is something we genuinely rely on. Generative agents are judged on believability by human raters;Park et al., 2023 (Generative Agents) — memory/reflection/planning agents whose success is "believability," judged by humans, not a checker. Park et al., 2024 built agents from interviews that replicate individuals' survey responses ~85% as well as the people replicate themselves. social agents in SOTOPIA are scored on seven subjective dimensions with free-text rationales;Zhou et al., 2023 (SOTOPIA) — goal-oriented social interaction scored on seven dimensions (believability, relationship, knowledge, secret-keeping, social rules, financial/material benefit, goal). The rationales are inherently verbal. (One of us is an author here — this is the kind of task that motivated the whole line of work.) theory-of-mind benchmarks reveal that scalar accuracy can hide illusory competence, a right answer for the wrong reason.Kim et al., 2023 (FANToM) shows interactive, information-asymmetric ToM where high accuracy can mask shallow reasoning — "illusory theory of mind." Wu et al., 2023 (Hi-ToM) pushes to higher-order recursive beliefs. A scalar cannot tell you the model got it right for the wrong reason; a verbal critique can.

To measure this, Ditto introduces SOUL (Simulation gym Of hUman-Like behavior): 10 tasks across six categories — Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation. These are deliberately the tasks where a scalar reward is least informative.

Trained with Ditto, an 8B model improves an average of 36% over the base policy (0.533 → 0.726) and beats GPT-5.4 on 6 of the 10 tasks, despite being far smaller. The average is the least interesting number, though — what matters is where the gains land.

The blowouts are on Social Skill (Sotopia, 0.28 → 0.47) and User Simulation (UserLLM, 0.47 → 0.93; MirrorBench, 0.55 → 0.71) — the most interactive, most multi-dimensional tasks on the board. GPT-5.4 still leads on the role-play tasks that lean more on stored knowledge than on moment-to-moment social calibration. That split is the thesis: verbal feedback buys almost nothing where the scalar was already honest, and almost everything where it was hiding the most.

A few honest caveats

We find the arc compelling, but it is worth being clear about what verbal feedback does not buy you.

The feedback here comes from an LLM judge, not a human. That puts Ditto in the same family as RLAIF: the policy can only become as nuanced as the critiques it trains on, so the judge's blind spots become the policy's blind spots. It also opens a language-space version of reward hacking — verbosity, sycophancy, gaming the critic's stylistic tells — that we are only beginning to understand.

It is not free, either. Each prompt now needs two rollouts — student and feedback-conditioned teacher — so you pay roughly double the generation cost per step. The bet is that the better target is worth more than the extra samples; on SOUL it clearly was, but that is workload-dependent.

And internalization cuts both ways. Because the feedback dimensions get baked into the weights, you cannot cheaply change your mind about what you want at test time the way you can by re-prompting a system that reads feedback live (the Reflexion trade, in reverse). If the target moves, you retrain.

Finally, beating GPT-5.4 on SOUL is not the same as being indistinguishable from a person — and the illusory-ToM worry applies to our own model too. The real test of verbal feedback is whether the student internalized the reasoning the critique pointed at, or just the answer. That is the thing we most want to verify next.

Coda: R is a sentence now

Return to the gradient we started with, $\nabla_\theta\, \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$. Across the entire arc — RLHF, RLAIF, RLVR, and now verbal feedback — that gradient barely changes. What changes is the thing we put inside the expectation. For a decade it was a scalar, because a scalar is what made the gradient easy. The arc is the field slowly admitting that the native form of human feedback was always language, and engineering its way back to it.

No coach hands an athlete a 0.62. They say you're late on the backhand. The interesting frontier in RL for language models is the same correction, finally arriving: the reward has gotten the bandwidth of a sentence.

@misc{zhou2026verbal,
  author = {Xuhui Zhou and Weiwei Sun},
  title = {Learning with Verbal Feedback},
  year = {2026},
  howpublished = {\url{https://xuhuizhou.com/blog/learning-with-verbal-feedback}},
}