DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction • Zhancun Mu

Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Zhancun Mu and Chi Zhang, School of Intelligence Science and Technology, Peking University.

Latest manuscript (PDF) · arXiv record ↗ · Code ↗

Learning From Data Without Trusting Every Prediction#

Offline reinforcement learning learns a decision-making policy from a fixed dataset, without collecting new interactions during training. Its central challenge is to improve beyond recorded behavior without exploiting overly optimistic value estimates for unfamiliar actions. Simply imitating the data limits improvement; maximizing predicted value without sufficient regularization can produce unreliable actions.

Expressive generative models can represent multiple plausible actions for the same state. DeFlow studies how to extract a useful policy from a learned multi-step flow prior while keeping value improvement separate from behavior-prior modeling.

Method: Propose, Then Refine#

The base flow proposes an action from the learned behavior distribution. An action-conditioned residual network then makes a local adjustment:

a_{\mathrm{base}} = \operatorname{sg}[\mu_\psi(s,z)], \qquad a = \operatorname{clip}\bigl(a_{\mathrm{base}} + f_\phi(s,a_{\mathrm{base}})\bigr).

The stop-gradient operator prevents value-improvement gradients from propagating through the flow solver. The prior is trained through flow matching, while the residual learns from the critic. Conditioning on the proposed action lets different proposals at the same state receive different corrections.

An adaptive penalty discourages large residuals. The task-specific budget $\delta$ controls the mean squared adjustment per action coordinate; the penalty weight $\alpha$ increases when that empirical budget is exceeded and decreases otherwise. This is a soft regularizer around the learned prior, not a guarantee that actions remain within the true dataset support. The budget still requires selection.

Results and Scope#

We evaluate DeFlow on 73 offline RL tasks and 15 offline-to-online tasks. It is competitive with recent generative methods, with clearer benefits on selected manipulation tasks, including cube-double, puzzle-3x3, and visual manipulation. It is not uniformly stronger: shortcut and inference-scaling approaches such as SORL remain strong on several navigation and scene tasks, and the SORL comparison is not compute-matched. The latest manuscript includes the aligned SORL comparison and full task-level results.

Action landscapes showing base-flow proposals, one-step actions, and DeFlow refinements.

These visualizations illustrate local refinement around proposals from the learned flow. They are qualitative diagnostics, not proof of behavior-support preservation or a general failure of one-step policies.

Offline-to-online learning curves on OGBench, D4RL antmaze, and D4RL adroit.

In the reported offline-to-online protocol, we reuse the offline settings rather than retuning the residual budget for online adaptation. Results vary across tasks and depend on the available data and learned prior.

Additional Diagnostics and Limitations#

The revised manuscript includes residual-capacity sensitivity across nine tasks, prior-only comparisons at shared checkpoints, residual-budget sensitivity, and runtime analysis. Smaller residual networks can be competitive or better on some tasks, but there is no universal best depth. Reduced exploitation of critic errors is a possible explanation, not an established causal finding.

DeFlow cannot reliably recover useful actions missing from its learned prior. Adaptive regularization does not eliminate hyperparameter choices or critic errors, and inference still requires multi-step flow sampling. The fixed-penalty diagnostic also carries an explicit implementation qualification in the manuscript; it should not be read as a fully calibrated coefficient sweep.

Acknowledgments and Disclosures#

Zhancun Mu thanks his former advisor, Yitao Liang, for his support. This work was supported by the Shanghai Artificial Intelligence Innovation Center under Grant No. 2025SHZDZX026D22. The authors declare no competing interests relevant to this work.

Citation#

@inproceedings{mu2026deflow,
  title={DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction},
  author={Mu, Zhancun and Zhang, Chi},
  booktitle={Advances in Neural Information Processing Systems},
  year={2026}
}

bibtex

Abstract