Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

公司动态 Hugging Face 原文语言:英文

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

全文

TL;DR -- For those of you who don't have time to read 5,000 words about async RL plumbing (we get it, you have models to train):

The problem: In synchronous RL (reinforcement learning) training, data generation (model inference to create data samples) dominates wall-clock time -- a single batch of 32K-token rollouts on a 32B (32-billion parameter) model can take hours, while the GPUs used for training remain idle.

The solution everyone converged on: Disaggregate (separate) inference and training onto different GPU pools, connect them with a rollout buffer (temporary storage for model outputs), and transfer weights asynchronously (without waiting), so neither side waits for the other.

We surveyed 16 open-source libraries that implement this pattern and compared them across 7 axes: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, and distributed training backends.

Key findings: Ray dominates orchestration (8/16 surveyed distributed computing libraries). The NCCL (NVIDIA Collective Communications Library) broadcast is the default method for transferring model weights. Staleness management refers to how outdated data samples are handled, ranging from simply dropping old samples to using advanced importance-sampling correction. LoRA (Low-Rank Adaptation) training is sparsely supported. Distributed MoE (Mixture of Experts) support is the emerging differentiator.

If you'd rather skip straight to the good part, here's the full comparison table (no reading required, we won't judge).

But seriously, if you stick around, you might learn a thing or two about why your GPUs are idle 60% of the time.

Click to expand Table of Contents

1. Motivation: From synchronous RL training to async architectures

1.1 How TRL Does RL Training Today

1.2 Colocated vs. Disaggregated Training

1.3 The Generation Bottleneck

1.4 The Core Insight

2. Libraries Surveyed

3. The Comparison Framework: Seven Axes

Axis 1: Orchestration & Concurrency Primitive

Axis 2: Rollout Buffer Design

Axis 3: Weight Synchronisation Protocol

Axis 4: Staleness Management

Axis 5: Partial Rollout Handling

Axis 6: LoRA Training Support

Axis 7: Distributed Training Backend & Parallelism

4. Global Overview: Sixteen Libraries at a Glance

5. The Next Wave: Design Implications

5.1 Critic-Free Algorithms: Memory Freed, But Weight Sync Pressure Increases

5.2 Process Rewards: A New Synchronisation Barrier

5.3 Multi-Agent Co-Evolution: The Straggler Problem Compounds

5.4 Training-Inference Mismatch: The Deepseek v3.2 MoE Case Study

5.5 Distillation: The Same Async Problem Under a Different Name

6. Design Choices for TRL's Async Trainer

Design Principle: Keep Orchestration Lightweight

1. Bounded Queue with Per-Token model_version (No Double-Buffering)

2. NCCL Weight Sync with Packed Transfers

3. Partial Rollout Support for Agentic Workloads

1. Motivation: From synchronous RL training to async architectures

Async RL training has emerged as the dominant paradigm for post-training at scale. Several trends in modern post-training have made synchronous training loops nearly impossible to scale:

Long rollouts from reasoning models. Chain-of-thought training produces very long rollouts, and a single synchronous generation batch can take hours to complete on a single GPU. During all of that time, training GPUs sit completely idle.

Value-function-free trainers like GRPO use group-relative advantages. This means generating up to G times more rollouts per prompt, and the entire batch is gated by the slowest completion in the group.

The rise of agentic RL training. When models interact with tools, sandboxes, and external environments across multi-turn trajectories, rollout lengths and latencies become highly variable. A simple API call might return in seconds, while a complex reasoning chain with tool use can run for minutes or hours. MiniMax's Forge framework, used to train MiniMax-M2.5, illustrates the scale this reaches in practice: context lengths up to 200K tokens, over a hundred thousand distinct agent scaffolds and environments, and daily throughput on the order of millions of samples. At this scale, any synchronous barrier between generation and training becomes a severe bottleneck. The straggler problem alone (where a handful of slow rollouts block an entire batch) can idle hundreds of GPUs.

The open-source ecosystem has converged on a common architectural response: disaggregate inference from training onto separate GPU pools, connect them with a rollout buffer, and let both sides run concurrently.

We are developing a new async trainer for TRL, one of the most widely used libraries for model post-training. To guide our design, we surveyed sixteen open-source libraries that were built from the ground up around asynchronous training and compared them across seven axes: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, and distributed training backends. This article distills the design principles we extracted from that survey.

Beyond RL, the need for async infrastructure is increasingly evident. For example, on-policy distillation, where a student generates sequences and a teacher scores them, mirrors GRPO but swaps the reward function for a teacher forward pass. Recognizing this structural similarity, everything in this survey applies equally to async distillation. We'll return to this broader point in Section 5.

1.1 How TRL Does RL Training Today

TRL's current GRPOTrainer implements the full GRPO loop (prompt sampling, generation, reward scoring, advantage computation, gradient update, and weight sync) in a single synchronous training_step() call. This design is simple and correct, but it cannot overlap generation with training, leaving significant GPU utilisation on the table.

Looking at the GRPOTrainer, we have the following phases sequentially within each training step:

Prompt sampling: draw a batch of prompts from the dataset. Nothing crazy here, let's continue.

Generation, calls model.generate() (or forward requests to a vLLM server) to produce G completions per prompt. This is autoregressive and dominates wall-clock time.

Reward scoring: evaluate each completion against one or many reward functions.

Advantage computation

Forward and backward passes: compute the clipped policy gradient loss and backpropagate.

Optimizer step, update model weights.

Weight sync, push updated weights to the inference engine (vLLM) so the next generation uses the new policy.

Each phase blocks until completion before the next begins. The timeline looks like this:

TRL offers the steps_per_generation config option to reuse a single set of rollouts across multiple gradient steps (temporal reuse), amortizing the generation cost. But the generation call itself remains fully synchronous and blocking; the trainer cannot begin gradient computation until every completion in the batch has finished.

The library also supports running vLLM in server mode as a separate process. It frees the training GPU during generation, but two hard synchronisation barriers remain: the HTTP calls until all completions return, and the weight sync blocks both the trainer and vLLM during the transfer.

1.2 Colocated vs. Disaggregated Training

Before discussing async training, it is essential to understand the two deployment topologies for RL training with a separate inference engine:

Colocated mode places inference and training on the same set of GPUs. A single GPU (or TP group) holds both the training model (under FSDP or ZeRO) and the inference engine (vLLM or SGLang). Only one role is active at a time: during generation, the training model's parameters may be offloaded or resharded into an inference-friendly layout (e.g., from FSDP shards to vLLM's tensor-parallel layout); during training, the inference engine is paused or put to sleep. Weight "sync" is essentially free; it is at most an in-place resharding on the same GPU, not a network transfer. The advantage of the colocated mode is simplicity and cost; you need fewer total GPUs. The fundamental limitation is that inference and training cannot overlap. For example, here is the Trl with vllm in colocate_mode:

Disaggregated mode places inference and training on separate GPU pools. The inference pool runs vLLM or SGLang continuously; the training pool runs the optimizer continuously. The two pools communicate via a weight synchronisation protocol (NCCL broadcast, filesystem checkpoint, HTTP, etc.) and a data transfer mechanism (Ray object store, Redis streams, shared memory, etc.)

The biggest advantage of the disaggregated mode is that inference and training can run concurrently. While the trainer computes gradients on batch N, the inference pool is already generating rollouts for batch N+K, enabling async training. However, this benefit comes at a cost: additional GPUs are required.

Concurrency, asynchronicity, and parallelism are distinct concepts that often get conflated. In this article, when we say "async training," we mean something specific: generation and training running in parallel, with effective overlap; the inference pool is producing the next batch of rollouts while the training pool is computing gradients on the current batch. This is fundamentally a disaggregated-mode capability. Colocated mode can benefit from optimisations like sleep/wake memory management or fast in-place resharding to speed up inference, but it cannot achieve true simultaneous overlap; inference and training still take turns on the same GPUs. Every library in this survey that implements meaningful async overlap uses disaggregated mode as the foundation.

1.3 The Generation Bottleneck

In RL training for reasoning models, autoregressive generation dominates wall-clock time. A single rollout for a math or coding task can produce 8K–64K tokens of chain-of-thought reasoning (see QED-Nano rollout lengths).

To ground this concretely, consider vLLM benchmarks on a single H100 80GB GPU (bf16, no quantisation, offline throughput mode). A 7B model (DeepSeek-R1-Distill-Qwen-7B) achieves ~6,300 output tokens/s aggregate throughput; a 32B model (DeepSeek-R1-Distill-Qwen-32B) drops to ~1,200 output tokens/s. These are total throughput across all concurrent requests, the number the inference engine can push through per second, regardless of how many sequences share the GPU.

Now consider a typical GRPO training step: G=8 completions per prompt × 64 prompts/batch = 512 rollouts. How long does generation take?

Output length per rolloutTotal output tokens (512 rollouts)Time on 1×H100 (7B @ ~6K tok/s)Time on 1×H100 (32B @ ~1.2K tok/s)

2K tokens (short CoT)~1M tokens~3 min~14 min

8K tokens (medium CoT)~4M tokens~11 min~56 min

32K tokens (long CoT)~16M tokens~45 min~3.7 hours

Even at the short end (2K tokens generated with a 7B model), generation alone consumes several minutes per training step. At the long end, where frontier reasoning models increasingly operate, a single generation phase can take hours on one GPU. Scaling to 8 inference GPUs divides these times by roughly 8× (assuming linear throughput scaling), but even then, 32K-token rollouts on a 32B model still take ~28 minutes per step.

The straggler problem compounds this further. In group-based algorithms like GRPO, you sample G completions per prompt. The batch cannot proceed until the slowest completion finishes. Chain-of-thought output lengths are highly variable; a single prompt might produce completions ranging from 1K to 32K tokens. The batch is gated by the longest completion, and continuous batching only partially mitigates this: shorter sequences free up slots for new work, but the last sequence in a GRPO group still blocks the group's reward computation and training step.

1.4 The Core Insight

Every library in this survey has independently converged on the same architectural principle: physically separate inference GPUs from training GPUs, and push weights asynchronously, so generation never stops and training never waits.

The inference pool runs continuously, feeding completed rollouts into a buffer. The training pool pulls from the buffer, computes gradient updates, and periodically pushes new weights back to the inference pool to keep it in sync. The two loops run at their own pace, decoupled by the buffer.

This setup is highly scalable, but it introduces a new class of problems: staleness (rollouts generated under an old policy), weight synchronisation overhead, partial rollout handling, etc. The rest of this article dissects in detail how current open-source libraries address these issues.

2. Libraries Surveyed

LibraryOrganisationRepoGitHub ⭐ (Mar. '26)

AReaLinclusionAI/Ant Groupgithub.com/inclusionAI/AReaL4,338

ARTCoreWeavegithub.com/OpenPipe/ART8,952

AtroposNousResearchgithub.com/NousResearch/atropos878

MILESradixarkgithub.com/radixark/miles950

NeMo-RLNVIDIAgithub.com/NVIDIA-NeMo/RL1,383

OATSAIL-SGgithub.com/sail-sg/oat637

open-instructAI2 (AllenAI)github.com/allenai/open-instruct3,611

PipelineRLServiceNowgithub.com/ServiceNow/PipelineRL374

PRIME-RLPrimeIntellectgithub.com/PrimeIntellect-ai/prime-rl1,114

ROLLAlibabagithub.com/alibaba/ROLL2,921

SkyRLNovaSky-AIgithub.com/NovaSky-AI/SkyRL1,664

SLIMETHUDMgithub.com/THUDM/slime4,595

TorchForgeMetagithub.com/meta-pytorch/torchforge632

TunixGooglegithub.com/google/tunix2,175

verlByteDancegithub.com/verl-project/verl19,673

verifiers-rlPrimeIntellectgithub.com/PrimeIntellect-ai/verifiers3,876

3. The Comparison Framework: Seven Axes

To make sense of the rapidly expanding ecosystem of async RL libraries, we propose seven orthogonal axes of comparison. Each axis captures a fundamental design decision that shapes the library's performance, complexity, and trade-offs.

Axis 1 – Orchestration & Concurrency Primitive: how distributed components are coordinated (Ray actors, asyncio, pub/sub, HTTP).

Axis 2 – Rollout Buffer Design: how rollouts flow from inference to training.

Axis 3 – Weight Synchronisation Protocol: how updated weights reach inference servers, and whether the system must pause to accept them or continue generating.

Axis 4 – Staleness Management: how off-policy rollouts are handled: version rejection, depth bounding, or importance-sampling correction.

Axis 5 – Partial Rollout Handling: what happens to in-flight generations when a weight update arrives mid-sequence.

Axis 6 – LoRA Training Support: General LoRA support and whether adapter-only parameters can be trained and synced, enabling sub-millisecond weight transfers.

Axis 7 – Distributed Training Backend & Parallelism: what parallelism strategy is used for training, constraining max model size.

Axis 1: Orchestration & Concurrency Primitive

How does the system coordinate its distributed components?

The choice of orchestration framework determines the programming model, failure semantics, and scalability ceiling. Rather than listing per-library implementation details, the landscape decomposes cleanly into four orchestration types, fundamental coordination paradigms that differ in abstraction level, failure model, and deployment requirements:

Orchestration TypeWhat It IsLibrariesTrade-offs

Distributed Actor ModelComponents are actors, isolated stateful processes with mailboxes, managed by a runtime that handles scheduling, resource placement, fault tolerance, and object transfer. Communication is via asynchronous RPC / futures / object store.Ray: verl, SkyRL, NeMo-RL, SLIME, MILES, ROLL, OAT, open-instruct. Monarch: TorchForge.Richest abstraction; solves scheduling and fault tolerance out-of-the-box. Adds a non-trivial runtime dependency and framework-specific debugging overhead.

Native Python ConcurrencyComponents are threads, coroutines (asyncio), threading primitives, multiprocessing child processes, and queues. No external orchestration runtime.verifiers-rl, PipelineRL (intra-pool), ART (asyncio + child-process proxies), AReaL (asyncio-based event loop)Minimal dependencies, easy to debug, full control. Limited to single-node unless paired with additional IPC (Redis, HTTP, NCCL) for multi-node communication.

Pub/Sub Message BusComponents are decoupled producers and consumers communicating through append-only streams or message queues. Not orchestration per se, a data transport layer between independently running pools.PipelineRL (inter-pool: Redis XADD/XREAD streams for multi-node, append-only JSONL files for single-node)Clean decoupling across pool boundaries without RPC. Does not manage process lifecycle, scheduling, or fault recovery; must be paired with another orchestration type.

HTTP MicroservicesComponents are independent services communicating via REST APIs. Language-agnostic, maximum decoupling.AtroposAny inference server, any language, zero shared state. Highest latency (if NCCL); no shared object store; fault tolerance is the user's responsibility.

Note on Tunix: Tunix (Google) uses a JAX-native mesh model with ThreadPoolExecutor for async overlap and jax.device_put for cross-mesh weight transfer. It is architecturally distinct enough from the PyTorch ecosystem that direct comparison on orchestration is not meaningful; it lives in the XLA/TPU world with its own coordination primitives.

The table above reveals a striking pattern: eight of the sixteen libraries surveyed use Ray as their orchestration backbone. This is not a coincidence; it reflects a deep architectural fit between the actor model and the structure of RL training. A survey by Anyscale (the company behind Ray) of open-source RL libraries for LLMs confirms this convergence. RL training at large scales involves fundamentally heterogeneous components (inference engines, training engines, environments, reward models) that must be orchestrated across a cluster, often on different hardware types, with different scaling requirements and failure modes. Ray's actor model maps directly onto this:

Actor isolation and heterogeneous resources. Each RL component (vLLM inference server, FSDP trainer, reward model, environment pool) becomes a Ray actor with its own resource requirements (num_gpus, num_cpus, memory). Placement groups give fine-grained control over GPU affinity without manual SSH/torchrun orchestration.

Scheduling and autoscaling. Ray's scheduler handles the combinatorial problem of placing heterogeneous actors across a cluster. When generation requires 8× more GPU-hours than training, you can just tell Ray to scale your inference actors independently.

Fault tolerance. Long RL training runs (days to weeks) are vulnerable to GPU failures, OOM kills, and network partitions. Ray's actor restart policies and object store replication provide resilience that would require significant custom infrastructure with raw asyncio and multiprocessing. Concrete example of the fault tolerance: open-instruct, for example, relies on Ray's actor supervision to recover from vLLM engine crashes mid-rollout.

Object store for zero-copy data transfer. Rollout data can be large, tens of GB per batch for very long-context reasoning. Ray's shared-memory object store enables zero-copy transfer between actors on the same node, avoiding serialization overhead that usually comes with multiprocessing.Queue approaches.

Ecosystem maturity. Ray has been battle-tested at scale since 2017, with production deployments on thousands of GPUs. The debugging overhead is real (Ray Dashboard, distributed stack traces, placement group failures), but the alternative, building equivalent coordination from scratch, is worse at the multi-node scale. That said, Ray is a heavy dependency: it pulls in its own scheduler, object store, and dashboard, adding operational complexity that not every team needs. This is exactly why libraries like PRIME-RL, PipelineRL, and AReaL opted for lightweight native-Python coordination (asyncio, threading, Redis streams) instead --- when you control the full stack and your deployment topology is fixed, the simplicity and debuggability of vanilla Python often outweigh the conveniences Ray provides.

The cost is a hard dependency on a non-trivial runtime. This trade-off can be worthwhile, especially for production-scale training (64+ GPUs, multi-day runs, complicated reward computation).

While Ray's actor model is the main player on the field, Monarch emerged as a new PyTorch-native distributed actor framework from Meta, purpose-built for GPU workloads. Like Ray, Monarch is based on the actor model; components are independent actors with mailboxes communicating via messages, but it is designed from the ground up for the PyTorch/CUDA ecosystem rather than being a general-purpose distributed runtime.

Monarch offers several capabilities particularly relevant to async RL. An example implementation of async RL with Monarch (from the GPU Mode lecture series) demonstrates the architecture: generators, a replay buffer, and a trainer are modelled as Monarch actors, with the replay buffer absorbing latency variance from straggler rollouts and RDMA weight sync pushing updated parameters to generators without blocking training. The pattern is structurally identical to Ray-based designs (verl, SkyRL, open-instruct) but implemented with pure PyTorch-native primitives.

Axis 2: Rollout Buffer Design

How do generated rollouts flow from inference to training, and how deep is the pipeline?

The buffer is the data structure sitting between generation and training. Its depth controls the maximum degree of asynchrony, and therefore the maximum staleness.

PatternDepthLibrariesCharacteristic

No buffer (synchronous)0TRL (current), ART (gather-all-then-train)Generation and training alternate strictly; zero staleness, maximum idle time

Double-buffer (one-step-ahead)1verifiers-rl, SLIME (async mode), MILES, OATSubmit generation N+1 at the start of training step N; overlap exactly one batch

Bounded async queue2–KSkyRL, verl (fully async), NeMo-RL, ROLL, PRIME-RL, TorchForge, Tunix, open-instruct (async_steps), AReaL (max_head_offpolicyness)Multiple batches in flight; staleness bounded by queue capacity

Unbounded / streamUnlimitedPipelineRL (Redis streams), SLIME (fully async mode), AtroposContinuous generation; staleness bounded only by explicit version control

The double-buffer pattern is the simplest upgrade from synchronous to asynchronous training: it overlaps exactly one generation with one training step and introduces at most one step of policy lag !

Deeper queues, on the other hand, improve throughput but require staleness management.

The buffer controls how much data is in flight. But data is only half the equation. The other half is getting updated weights back to the inference servers before those rollouts go stale. That's where weight sync comes in!

Axis 3: Weight Synchronisation Protocol

How do new model weights reach the inference servers after a gradient update?

Scope note: This axis focuses on disaggregated mode, where inference and training run on separate GPU pools, since that is the deployment topology where async overlap (and therefore weight sync design) actually matters. Colocated setups (same GPUs for both roles) are inherently synchronous and do not face the transport/interrupt trade-offs discussed below.

This is the most architecturally consequential axis. The protocol determines sync latency, interrupt granularity, and whether partial rollouts are possible.

There is a critical distinction to make here: the transport mechanism and the interrupt model. Most libraries pause generation at a coarse boundary, an HTTP request, a full batch, or even a full training step, before initiating weight transfer. PipelineRL is the outlier: it never stops generating at all.

Transport mechanism:

MechanismLatencyLibraries

NCCL Broadcast~100–500msPipelineRL, SkyRL, SLIME, MILES, ROLL, OAT, NeMo-RL, PRIME-RL, open-instruct, AReaL

NCCL + Bucketing~20msverl

KV + Shared MemoryLowTorchForge

Filesystem + HTTPMediumPRIME-RL, AReaL, ART

CUDA IPC (Zero-copy)Very LowNeMo-RL, MILES

JAX Cross-meshLowTunix

HTTP PUTHighverifiers-rl

Filesystem + RestartVery HighAtropos

In the interrupt model, when does the generation pause to accept new weights?

This is where PipelineRL fundamentally diverges from every other library. Rather than listing each library individually, the landscape collapses into five conceptual tiers, ordered from finest to coarsest interrupt granularity:

Interrupt GranularityWhat HappensLibraries

Never (In-flight per-forward-pass)Sequences never stop. The weight swap happens between token decode steps (~1-10ms gap). Running sequences seamlessly continue with new weights.PipelineRL, open-instruct (opt-in)

Per HTTP Request (Abort + Resync)In-flight HTTP requests are aborted. Partial tokens are resubmitted with a prefix-resume mechanism or recycled for retry.SkyRL, SLIME, MILES

Soft Pause (Drain in-flight)No new generation requests are accepted while in-progress ones finish naturally. Once drained, weights are synced and generation resumes.PRIME-RL, AReaL, open-instruct (default), verl (async)

Per Training Step / Batch (Blocking)Generation must fully complete. The trainer and inference engine take turns blocking each other.NeMo-RL, ROLL, OAT, TorchForge, Tunix, verifiers-rl, Atropos

The "never-stop" tier is qualitatively different from all others: PipelineRL, for example, hooks into the inference engine so that the lock is acquired and released per transformer forward pass (one token step for one sequence). A weight update waits at most one forward pass (~few ms), swaps all parameters, and generation resumes immediately. Every other library stops generation at a coarser boundary, from one HTTP request (~hundreds of ms) up to a full batch boundary (~seconds).

Weight sync controls when new weights arrive. But async training means rollouts are always being generated under some policy version, and that generating policy might be several gradient steps behind the trainer. How libraries handle this policy lag is staleness management.

Axis 4: Staleness Management

How does the system handle the fact that generated rollouts may come from an older policy than the one being trained?

Once generation and training overlap, samples become off-policy. Three orthogonal strategies have emerged for managing this staleness, and most production systems combine more than one:

Strategy 1: Per-sample version rejection. Every sample is tagged with the integer policy version that generated it. At training time, samples whose version falls behind the current policy by more than a threshold are hard-dropped before entering the loss computation. Simple and correct, but wastes the precious compute spent generating discarded samples.

Strategy 2, Depth Bounding. The queue or buffer between generation and training has a bounded capacity (or an explicit staleness gate), which architecturally limits how far behind any sample can be. This ranges from depth=1 (one-step-ahead double buffering, where staleness is impossible by construction) to explicit capacity formulas tied to version gaps. No per-sample version tracking is required; the bound is enforced by the system's pipeline depth.

Strategy 3, IS-weighted loss correction. Stale samples that reach the trainer are reweighted by the importance sampling ratio π∗θ(a∣s)π∗old(a∣s)\frac{\pi*{\theta}(a \mid s)}{\pi*{\text{old}}(a \mid s)}π∗old(a∣s)π∗θ(a∣s)​, typically clipped (Truncated IS). Some libraries also apply OPSM (zero-out loss for off-policy samples with negative advantage). This preserves throughput; no samples are discarded, but there is a cost in gradient variance from the IS ratios.

These strategies are orthogonal: a system can use version rejection alone, depth bounding alone, IS correction alone, or any combination of them. Synchronous systems avoid the problem entirely by never overlapping generation and training.

LibraryVersion RejectionDepth BoundingIS CorrectionKey Config / Notes

AReaL❌✅⚠️max_head_offpolicyness capacity formula; optional use_decoupled_loss adds IS weight capped at 5.0

ART———Synchronous; all rollouts collected before training; no staleness by design

Atropos❌✅❌max_batches_offpolicy, ceiling on buffered batches

MILES❌❌✅TIS + OPSM

NeMo-RL✅❌❌max_trajectory_age_steps, per-sample version drop

OAT❌❌✅Clipped TIS ratio

open-instruct❌✅⚠️async_steps cap (default 1, production 8); optional --truncated_importance_sampling_ratio_cap ρ adds clipped TIS

PipelineRL✅❌❌max_lag, integer version tag per sample; drop if age exceeds threshold

PRIME-RL✅✅✅Full hybrid: max_async_level version gap + max_off_policy_steps cancellation + IPO trust-region IS

ROLL❌❌✅Richest IS suite: TIS, TOPR, CISPO, Kimi15, six off-policy loss variants

SkyRL❌✅❌max_staleness_steps, capacity gate blocks new rollouts when exceeded

SLIME❌❌✅TIS + OPSM (off-policy masking for partial rollouts)

TorchForge✅❌❌max_policy_age, per-sample version tag; hard drop

Tunix❌✅❌Bounded queue + sync per step; staleness structurally limited

verl❌❌✅Clipped TIS ratio; optional OPSM

verifiers-rl❌✅❌Depth=1 FIFO + sync every step; staleness impossible by construction

✅ = yes, ❌ = no, ⚠️ = optional / configurable, — = not applicable (synchronous)

Version rejection is simple and correct, but wastes compute when many samples are discarded.

IS correction preserves throughput at the cost of gradient variance.

Depth bounding is the coarsest mechanism, but it avoids per-sample bookkeeping entirely.

The trend in production systems (PRIME-RL, AReaL, open-instruct) is toward hybrid approaches that combine depth bounding with optional IS correction, getting the architectural simplicity of bounded queues with the loss-level safety net of importance weighting for stable training.

Staleness management handles data that was generated under an old policy. But what about data that's still being generated when a weight update lands?

Axis 5: Partial Rollout Handling

What happens to a generation in progress when a weight update arrives?

This is critical for long-context tasks where a single rollout can take minutes. Four strategies:

StrategyLibrariesDescription

Implicit continuationPipelineRLSequences are never interrupted. Weights swap between forward passes; the sequence simply continues with new weights. Stored logprobs remain valid because training uses the recordedπ_old\pi\_{\text{old}}π_old, not recomputed.

Abort + retry with prefixSkyRL, SLIMEActive sequences are aborted. Partial tokens are accumulated, then resubmitted with a prefix-resume mechanism using the new weights.

Explicit save/resumeverl (fully async)The rollout worker saves partial token IDs and logprobs to a buffer, waits for sync, then resumes from the saved prefix.

Group cancellation (generation continues)PRIME-RLStale rollout groups have their async tasks cancelled; the inference server continues serving in-flight HTTP requests whose results are discarded. Weight sync triggers between HTTP requests without interrupting mid-request generation.

No partial rollout supportverifiers-rl, OAT, Atropos, TunixWeight sync only happens at batch boundaries. In-flight generations must complete before sync begins.

Soft pause, in-flight sequences completeAReaLA pause signal blocks new KV-cache allocations but does not abort in-progress sequences. The task dispatcher stops submitting new tasks; running tasks run to completion. After weight sync, generation dispatch resumes.

Full sleep, no in-flight at sync timeARTBy design, training only begins after all rollouts are collected. There are never in-progress sequences when sleep is triggered. Level-1 sleep (in-progress requests exist) offloads KV cache to CPU; level-2 sleep discards it entirely.

Drain-or-inflight (configurable)open-instructDefault: a stop flag gates new prefetching; weight update waits for active tasks to drain. With in-flight updates enabled, drain is bypassed and weights broadcast while tokens are still being generated; sequences in progress continue with a mix of old and new weights.

So far, every axis has assumed full-parameter training. But in LoRA training, you're only training a few million adapter parameters instead of billions, the weight sync problem nearly disappears. Let's look at how these libraries support LoRA training.

Axis 6: LoRA Training Support

Does the library support parameter-efficient training via LoRA adapters, in what modes, and does it exploit adapter-only weight sync?

LoRA is arguably the most practically consequential axis for teams with limited GPU budgets. It reduces the trainable parameter count by 99%+, halves peak activation memory, and, when the inference server is LoRA-aware, enables adapter-only weight sync: instead of broadcasting every parameter of a 7B+ model (~100–500ms NCCL), only the adapter deltas are pushed to vLLM, which at rank 32 amounts to ~50 MB, a sub-millisecond transfer.

LibraryLoRA SupportedMode RestrictionLoRA BackendAdapter-Only Sync

AReaL✅ YesFSDP2 only (not Megatron/Archon)HF peft✅ Yes (disk-based sync; only trainable params transferred; vLLM adapter hot-swap)

ART✅ Yes (primary design)Both (shared + dedicated GPU)Unsloth/peft (default); custom Megatron LoRA✅ Yes (only adapter saved/loaded; in-process or HTTP adapter hot-swap; base weights never moved)

Atropos✅ YesDisaggregatedHF peft✅ Yes (lora_only / lora_restart modes)

MILES✅ YesBoth (colocated + disaggregated)Megatron-Bridge✅ Yes (adapter sync config for SGLang)

NeMo-RL✅ Partial*BothCustom (not peft)❌ No evidence

OAT✅ YesBothHF peft✅ Yes (LoRA-only sync mode)

open-instruct⚠️ Code exists, not wired‡—HF peft (SFT/DPO only)❌ No (LoRA not applied in the RL trainer)

PipelineRL✅ YesNon-colocatedHF peft❌ No (full NCCL broadcast)

PRIME-RL✅ YesDisaggregatedCustom MultiLoRA (not peft)✅ Yes (adapter-only state dict extraction)

ROLL✅ Partial†DeepSpeed backend onlyHF peft / TRL❌ No evidence

SkyRL✅ YesBothpeft (FSDP) / Megatron-Bridge (Megatron)✅ Yes (filesystem-based adapter sync)

SLIME❌ No——❌ No

TorchForge❌ No——❌ No

Tunix✅ YesBothqwix (JAX-native)✅ Yes (auto-detected)

verl✅ Yes (most complete)Bothpeft (FSDP) / Megatron-Bridge (Megatron)✅ Yes (unmerged adapter sync)

verifiers-rl✅ Yes (via prime-rl)DisaggregatedHF peft + FSDP2 + vLLM✅ Yes (vLLM LoRA serving)

* NeMo-RL: LoRA for GRPO and DPO is supported only on the DTensor backend; the Megatron Core backend is SFT-only (RL LoRA listed as "coming soon"). Uses a custom DTensor-compatible LoRA module (not peft), optionally with Triton kernels.

† ROLL: LoRA is officially supported with the DeepSpeed training backend only. Megatron-backend LoRA appeared in the Feb 2026 changelog but remains experimental.

‡ open-instruct: The model config exposes LoRA-related fields (use_peft, lora_r, lora_alpha), and adapter saving is handled in the checkpoint logic. However, the peft model is never initialised in the RL training path; LoRA remains an SFT/DPO-only feature for the RL trainer as of March 2026.

Three LoRA implementation families:

HuggingFace peft (PipelineRL, SkyRL/FSDP, verifiers-rl, ROLL, OAT, Atropos): The most common choice. Standard checkpoint format (adapter_model.safetensors), compatible with any HF Transformers training loop. ZeRO-3 interactions require care: OAT, for example, needs to disable the fused LM head; ROLL must disable gradient checkpointing entirely.

Megatron-Bridge (verl/Megatron, SkyRL/Megatron, MILES): Required for 3D-parallel training (TP × PP × DP). Supports multiple LoRA types: lora, canonical_lora (splits merged QKV → separate Q/K/V adapters), vlm_lora, and dora. The canonical_lora variant avoids the QKV merge, thereby improving training stability. MILES saves checkpoints in both HF peft format and Megatron-native per-rank format.

Custom implementations (NeMo-RL, PRIME-RL, Tunix/qwix): Library-specific LoRA modules not interoperable with peft checkpoints. PRIME-RL uniquely supports multiple simultaneous adapters in a single run for multi-experiment parallelism. Tunix uses Google's qwix JAX library, which adds built-in QLoRA (NF4 quantization) and TPU-native gradient routing. NeMo-RL uses a custom DTensor-compatible module with an optional Triton fused kernel.

The adapter-only weight sync opportunity (interaction with Axis 3):

Eight of the thirteen libraries support pushing only the LoRA adapter deltas to the inference server. This changes the character of the weight sync problem (Axis 3) entirely. When using full-parameter training, the interrupt model (per-forward-pass lock vs. per-request abort vs. per-batch pause) determines how much generation is wasted during an NCCL broadcast. When using LoRA with adapter-only sync, the transfer is so small that nearly any interrupt model delivers equivalent throughput! Even Atropos's brute-force HTTP hot-swap becomes viable.

Axis 7: Distributed Training Backend & Parallelism

What parallelism strategy does the library use for training, and how does this constrain or enable the async architecture?

This axis cuts across every other axis. The choice of training backend determines how large a model can fit per GPU, how many collective operations are needed to gather weights before broadcasting to inference servers, and which model architectures can be trained at all. It is the single most consequential decision for teams scaling beyond 30B parameters or moving from dense to Mixture-of-Experts models.

LibraryTraining BackendParallelismHF Model LoadingMoE / EP Support

AReaLFSDP2, Megatron, ArchonDP, SP, TP, PP, CP, EP✅ Direct / Convert✅

ARTUnsloth, MegatronDP, TP, EP✅ Direct / Convert✅

AtroposPyTorch Native, TRLDP✅ Direct❌

MILESMegatron, FSDP2DP, TP, PP🔄 Convert✅

NeMo-RLFSDP2, MegatronDP, SP, TP, PP, CP, EP✅ Direct / Convert✅

OATDeepSpeedDP, TP✅ Direct❌

open-instructDeepSpeedDP, SP✅ Direct❌

PipelineRLDeepSpeedDP, SP✅ Direct❌

PRIME-RLFSDP2DP, TP, CP, EP✅ Direct✅

ROLLDeepSpeed, Megatron, FSDP2DP, SP, TP, PP, CP, EP✅ Direct / Convert✅

SkyRLFSDP, MegatronDP, SP, TP, PP, EP✅ Direct / Convert✅

SLIMEMegatronDP, TP, PP, SP🔄 Convert✅

TorchForgeFSDP2DP, TP, CP✅ via TorchTitan❌

TunixJAX/XLADP, TP❌ Custom Flax❌

verlFSDP, MegatronDP, SP, TP, PP, CP, EP✅ Direct / Convert✅

verifiers-rlDeepSpeedDP✅ Direct❌

The training backend creates direct implications for async RL library design:

Weight sync speed is a direct function of the training backend, and faster sync means less staleness.

In a disaggregated async setup, weight sync does not necessarily stall inference. The key design decision is how the weight update interacts with in-flight generation; four strategies exist, ordered from least to most disruptive:

Atomic swap, no interruption. The full weight update is dispatched as a single blocking RPC to the inference engine. Each forward pass sees either all old or all new weights, never a mix. Generation pauses for at most one forward-pass gap (~few ms). (PipelineRL)

Per-parameter streaming, no interruption. Each parameter is sent as a separate RPC + NCCL broadcast. Forward passes interleave between individual parameter updates, so in-flight sequences genuinely see a mix of old and new weights across layers. Maximum overlap, but weakest consistency. (open-instruct, inflight mode)

Dispatch gate, drain in-flight, then sync. New requests are held back while in-progress sequences complete naturally; weights are broadcast only after the pipeline drains. No wasted tokens, but a sync bubble proportional to the longest in-flight sequence. (PRIME-RL, AReaL, open-instruct default, verl fully-async)

Hard pause or abort. Inference is paused, or in-flight requests are aborted before weight transfer begins. Cleanest consistency, highest wasted compute. (verl, SkyRL)

But even in libraries where inference continues, slower sync means longer periods where inference runs on stale weights. The policy version gap between the trainer and the inference pool grows with sync duration. Something to take i

正文以来源语言显示。

参考来源

← 返回每日简报