
Ep#117: Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning
Imitation learning has enabled amazing robot capabilities, empowered in part by diffusion or flow-matching policies. These generative models outperform direct action prediction, but come at a cost: they are often slower and harder to train. The performance gap is usually explained by better support of multimodality, but Yuchen Zhou and Jiayuan Mao say this is wrong: that instead, the advantage generative policies have over pure regression is better explained by state-depended variation and by heavier-than-usual Gaussian “tails.”
They propose HT-Policy, which learns an input-dependent residual scale and uses it to reduce the effect of these long-tail data example resulting in reduced influence of these long-tail examples on training, allowing you to train regression models that are competitive with flow matching.
To learn more, watch Episode #117 of RoboPapers, with Chris Paxton and Ruijie He!
Abstract
Learning from demonstration has enabled impressive robot behaviors. A common choice for policy learning is to use diffusion or flow matching (Flow-Policies), which often outperforms direct action regression trained with mean squared error (MSE-Policies). This gap is commonly attributed to multimodal demonstrations. We revisit this gap from the perspective of statistical modeling: how action-prediction residuals shape policy optimization. Our analysis of real-world robot demonstration data reveals substantial state-dependent variation in residual scales and heavier-than-Gaussian tails. While both MSE-Policies and Flow-Policies exhibit heavy-tailed action residuals, their training gradients behave differently: MSE allocates more gradient magnitude to observations with large action residuals, which hurts optimization. Motivated by these findings, we introduce heteroscedastic Student-t action regression (HT-Policies), which learns input-dependent residual scales and reduces the influence of heavy tails. HT-Policies predict action chunks with a single feed-forward pass and can reuse pretrained flow-matching-based policy networks as the backbone. Across four simulation benchmarks and real-robot evaluations, HT-Policies achieves success rates competitive with generative policy baselines, both when trained from scratch and from pretrained vision-language-action and world-action models, despite being faster in training and inference. Together, these findings shed light on the practical advantages of generative objectives in robot learning from demonstrations and offer an efficient direct-regression alternative for a range of architectures and tasks. Project page: this https URL
Learn More
Project Page: https://projects.thelabone.org/regression-policy/
ArXiV: https://arxiv.org/abs/2610.12231
Transcript
Edited for clarity and readability.
Introducing HT-Policies
Chris Paxton: Hey everyone, welcome to another episode of RoboPapers. Today we’re excited to have Yuchen and Jiayuan here to tell us about “Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning,” and HT loss. Could you tell us a little about yourselves?
Yuchen Zhou: Hello, I’m Yuchen Zhou, a first-year PhD student at the University of Pennsylvania, advised by Professor Jiayuan Mao. I’m working on robot learning and understanding the principles behind it.
Jiayuan Mao: My name is Jiayuan Mao. I’m an assistant professor at the University of Pennsylvania, working at the intersection of AI, machine learning, and robotics.
Chris: Tell us about the paper. What are you trying to do?
Jiayuan: I’ll give a brief introduction to the motivation, and then Yuchen can go into the technical details.
At a high level, we’re interested in understanding why certain robot learning techniques work. In this paper, we look at generative policies, including diffusion and flow-matching policies, which have become standard approaches for behavioral cloning.
By “understanding,” we don’t just mean a theoretical machine learning perspective. We want to understand these methods empirically: when we train them on real-world data and apply them to real tasks, what is actually happening? We want to go beyond broad assumptions about data distributions and task difficulty.
One earlier paper that motivated us was Minimal Iterative Policy, from researchers at CMU. They were studying a similar question: why do diffusion policies work?
The conventional explanation is that you need to model a multimodal distribution of actions. The classic example is Push-T: there’s a T-shaped object on a table, and you want to move it to a target location. You can push it from the left or the right, or choose different contact points. People think you need a diffusion policy to model those different possibilities.
One experiment in that paper took a pretrained diffusion policy, drew multiple samples, and averaged the actions. The usual assumption is that averaging actions from different modes should be bad: if some demonstrations go left and others go right, their average might not complete the task. But they found that averaging did not really hurt performance.
One intuition is that robot observations are very high-dimensional. If you look at a low-dimensional state, there may appear to be a stochastic choice between going left and right. With a high-dimensional image input, those situations may be distinguishable.
Another intuition is that ambiguity in action choice often appears only during a small part of an execution. At the beginning, you might choose whether to go left or right. Once you’ve made that choice, the rest of the policy can be effectively deterministic.
Minimal Iterative Policy still uses a diffusion-style denoising objective. We wondered whether there were simpler principles that could explain these observations. That motivated our study of real-world data and our ablations of different design choices.
Is multimodality really the main problem?
Yuchen: The typical explanation for diffusion policies is that robot datasets contain many multimodal action distributions. Under that explanation, an MSE regression policy fails because it averages across modes. If the demonstrations go left or right, the average goes forward, which may be the wrong action.
But we find that there isn’t as much multimodality in robot datasets as that explanation suggests. We think another reason regression policies perform worse than generative policies is a mismatch in the loss. The standard regression loss is mean squared error, or MSE, which can be derived from a Gaussian residual model.
Chris: This is really interesting because another thing we see empirically, including in papers on best practices for robot learning, is that even diffusion policies don’t necessarily handle multimodality very well. You often want extreme consistency across demonstrations. Hugging Face had a good blog post on clothes folding with observations along those lines. Is this related?
Jiayuan: Definitely. When we collect behavioral cloning data, we usually have a protocol for the data collectors: to do this task, follow these particular steps. The best practice is often to make the demonstrations as deterministic as possible.
In some early experiments, we used scripts to generate actions in simulation. Those actions were almost deterministic. Yet even when we controlled the data that carefully, fitting it with a diffusion model still worked better than direct L2 or MSE regression. We wanted to understand why.
RJ He: Would you say that multimodality is a subset of a broader robustness issue across demonstrations, and that this approach helps address that broader issue?
Jiayuan: That’s a slightly different angle from the one we take in the paper, but it’s related. One possible interpretation is that the training data contains long-tailed noise or outliers. If those points dominate training, the model may be less robust. At inference time, it can deviate from its intended path and perform badly. I think these perspectives could eventually be unified.
Yuchen: That also reminds me of a common way…
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit robopapers.substack.com
Weitere Episoden von „RoboPapers“



Verpasse keine Episode von “RoboPapers” und abonniere ihn in der kostenlosen GetPodcast App.








