arxiv:2510.06062

ASPO: Asymmetric Importance Sampling Policy Optimization

Published on Oct 7, 2025

· Submitted by

Runze Liu on Oct 8, 2025

Upvote

Authors:

Runze Liu ,

Abstract

ASPO addresses the imbalance in token weighting during OSRL by flipping Importance Sampling ratios and incorporating a soft dual-clipping mechanism, improving training stability and performance in LLMs.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Recent Large Language Model (LLM) post-training methods rely on token-level clipping mechanisms during Reinforcement Learning (RL). However, we identify a fundamental flaw in this Outcome-Supervised RL (OSRL) paradigm: the Importance Sampling (IS) ratios of positive-advantage tokens are mismatched, leading to unbalanced token weighting for positive and negative tokens. This mismatch suppresses the update of low-probability tokens while over-amplifying already high-probability ones. To address this, we propose Asymmetric Importance Sampling Policy Optimization (ASPO), which uses a simple yet effective strategy that flips the IS ratios of positive-advantage tokens, aligning their update direction with the learning dynamics of negative ones. AIS further incorporates a soft dual-clipping mechanism to stabilize extreme updates while maintaining gradient flow. Comprehensive experiments on coding and mathematical reasoning benchmarks demonstrate that ASPO significantly mitigates premature convergence, improves training stability, and enhances final performance over strong GRPO-based baselines. Our analysis provides new insights into the role of token-level weighting in OSRL and highlights the critical importance of correcting IS in LLM RL. The code and models of ASPO are available at https://github.com/wizard-III/Archer2.0.

View arXiv page View PDF GitHub 31 Add to collection

Community

RyanLiu112

Paper author Paper submitter Oct 8, 2025

Code: https://github.com/wizard-III/Archer2.0
Models & Data: https://huggingface.co/collections/Fate-Zero/archer20-68b945c878768a27941fd7b6
Zhihu: https://zhuanlan.zhihu.com/p/1950985242098799047

librarian-bot

Oct 9, 2025

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on HF Mirror checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Merakakrem

Apr 30

I was reading the OpenReview rebuttal and noticed an apparent inconsistency between Table 8 and Table 9 that I'd love to get the authors' clarification on.

Both tables seem to report the 8B actor + 8B critic configuration:

Table 8 (critic size scaling, presumably n=2 critics following the main setup): 8B actor + 8B critic = 74.7
Table 9 (number of critics, "using the 8B critic recommended in the previous evaluation", so n=2 means 2×8B critic): 8B actor + n=2 = 71.3
The 14B actor row matches exactly across both tables (83.4), which suggests the setups are intended to be identical — but the 8B actor row differs by 3.4 points.

A few questions:

Are the two configurations actually the same experiment? If Table 8 is n=1 instead of n=2, that would conflict with Table 9's n=1 = 67.6.
If they are the same, what causes the 3.4-point gap — different seeds, checkpoints, or eval snapshots?
If the gap comes from single-seed run-to-run variance, it would be comparable in magnitude to the ~3-point gain AsyPPO reports over GRPO on 8B/14B actors. Are there multi-seed results or error bars to confirm the reported improvements are above the noise floor?