Making One GPU Go Brrr: Reinforcement Learning in a Single CUDA Kernel
How we fit the whole RL loop for a language model (writing answers, grading them, training on them and shipping new weights) into one GPU kernel that never exits, and raced it against PyTorch on the same H100.
Table of contents
TL;DR
- The problem: RL for language models alternates between generating answers and training on them. Standard PyTorch has the CPU launch over a million tiny GPU operations to do this, and it stops generating every time it trains.
- The inspiration: Flapping Airplanes' demo of one GPU kernel that trains a model, serves it and plays Doom at the same time, and its open rebuild, open-flapping-airplanes. If one kernel can run three unrelated programs, it can run the two halves of RL.
- The system, MegaRL: one launch. 264 thread blocks on the H100's 132 SMs take numbered tasks from on-GPU queues. Four generation engines and a learner run at the same time, and the engines never stop.
- The RL change: answers may be a little stale. Each group of answers remembers which weights wrote it; the learner uses it only if it is at most 8 updates old, and a clipped importance ratio corrects for the difference. Not waiting for fresh answers makes each update 11× faster.
- The result: Qwen2.5-0.5B-Instruct learning four multiple-choice tasks. MegaRL generates 67.7k tokens/s: 11× HF PyTorch and 1.7× a hand-tuned PyTorch loop with CUDA graphs. It reaches 70% held-out accuracy after 1.5 s of training, against 8.6 s for HF PyTorch. All three end at the same accuracy within noise.
1. Goal
Reinforcement learning (RL) is how a language model learns from its own attempts: it answers a question, a program checks the answer, and the model is nudged toward the answers that scored well. It is also hard on hardware. The GPU spends most of its time writing answers one token at a time, and the usual training loop keeps interrupting that work to train.
I wanted to see how fast a single GPU could run the whole online RL loop if nothing ever left the GPU: sample answers, grade them, compute the update, publish the new weights, and keep going. Then I wanted to race it against PyTorch on the same GPU, with the same model, prompts and training step, and check that the faster system actually learns as well.
This post tells that story without the gory details: where the time goes in a normal loop, the project that inspired the design, how the kernel schedules its work, the small change to the RL algorithm that made it possible, and the race.
2. RL in one minute
We use GRPO (Group Relative Policy Optimization), a popular recipe for training language models with RL. For each question, the model writes four answers, called a group. A grader gives each answer reward 1 if its final Answer: X is correct and it ends properly, and 0 otherwise. Each answer's advantage is how much better it did than the group's average, divided by the group's spread. The update then makes positive-advantage answers more likely and negative ones less likely.
Our setup is small enough to fit on one GPU and quick to measure. The model is Qwen2.5-0.5B-Instruct, learning four multiple-choice tasks at once: generated arithmetic, ARC science questions, SciQ and CommonsenseQA. Answers are at most 64 new tokens, and each update trains on 16 groups (64 answers). Every result below is scored on 1,600 held-out questions (400 per task) that are never trained on.
Two things decide how fast this goes: how quickly you can produce answers, since generation is most of the work, and how long the generator sits idle while the trainer runs.
3. Where the time goes in a normal RL loop
Launch overhead. A program that runs on the GPU is called a kernel. PyTorch runs a model as a long sequence of small kernels, each launched by the CPU. For a small model writing one token at a time, each kernel finishes in microseconds, and the GPU spends much of its time waiting for the CPU to send the next one. Our straightforward Hugging Face (HF) PyTorch loop made 1,327,001 launches in 26.6 seconds, about 50,000 per second.
Stop-and-go. The usual loop generates for a while, stops generating to train, then resumes. A hand-tuned PyTorch loop fixes most of the launch overhead: it keeps 512 answers rolling, captures decoding in a CUDA graph and uses a static KV cache. That cuts launches to 253,525 and lifts generation to 40.1k tokens/s. But it still stops generating for every update: 179 times in the race, a median of 34 ms each, or 22% of the run.
So the goal became concrete: no CPU in the loop, and a generator that never stops.
4. The inspiration: a kernel that never exits
The idea came from a demo video by Flapping Airplanes: one GPU kernel that trains a model, serves it and plays Doom, all at the same time. Jerry Zhu (Shopify) and Arnold Xu (Magni) rebuilt it from the video, in the open, as open-flapping-airplanes. It is a persistent megakernel: one launch, and its blocks never exit. They pull training, serving and Doom tasks from queues in GPU memory until the run ends.
Watching it, the thought was simple. RL is just two programs, a generator and a trainer, that share a GPU and pass data back and forth. If one kernel can run a trainer, a server and Doom side by side, it can run the two halves of RL side by side, and the generator never has to stop. MegaRL's scheduler starts from open-flapping-airplanes' design and adds what RL needs: model versions, queues of graded answers, and rules for how stale an answer may be.
5. Scheduling 132 SMs from inside the kernel
An H100 has 132 streaming multiprocessors (SMs), the GPU's independent cores. MegaRL launches exactly two blocks of 256 threads on each SM, 264 blocks in all, once. Every block then loops: grab a task, run it, report that it is done, repeat, until the run ends.
Jobs, phases and tickets
The work is organized as jobs: four generation engines and one learner. A job advances in steps. One step of an engine produces the next token for each of its 128 answers; one step of the learner is one training update. A step is a fixed list of phases (normalize, project to queries/keys/values, attention, MLP, and so on, layer by layer), and each phase is split into many small tasks that can run in parallel on different SMs.
Every task gets a number, its ticket. A block takes the next ticket with a single atomic add, runs that task, then bumps the phase's completion counter. When a phase's counter is full, the next phase opens. Because a task only ever depends on tasks with lower tickets, whoever holds the lowest unfinished ticket can always make progress, so the scheduler cannot deadlock.
Who runs what
Each block has a home job. The first block on every SM belongs to a generation engine, and each engine owns a contiguous range of SMs. The second block on every other SM belongs to the learner; on the rest it is another generation block. That gives 198 generation blocks and 66 learner blocks.
One rule keeps every block busy: a block never takes a ticket it cannot run yet. While the learner waits for a complete batch of answers, its blocks run generation tasks instead of spinning. The matrix multiplies inside the tasks use Hopper's warpgroup tensor-core instructions (wgmma), so each task is fast, not just well scheduled.
6. What gets trained, and why the replay is exact
We train all of layer 13, one of the model's 24 transformer layers: its attention and MLP weights, 14,946,304 parameters in all.
The trained layer is used from the last prompt token onward, while the prompt itself always goes through the original weights. So each question's KV cache is computed once at setup and reused by every version of the model. While generating, the engines save the hidden state entering layer 13 for every token. The learner starts from those saved states (layers 0 to 12 never change), replays layers 13 to 23 with exactly the same arithmetic the engines used, and backpropagates from layer 23 down to 13. Because the replay matches, the learner's log-probabilities equal the generation-time ones bit for bit when the weights have not changed, so any difference it sees is real policy change, not numerical noise.
Each update clips the gradient to norm 1.0, applies AdamW on full-precision master weights, and publishes a 29.9 MB bf16 snapshot of the new layer for the engines to use. In the race, an update took about 65 ms of computing plus about 37 ms of waiting for a complete batch, time the learner's blocks spent helping generate.
7. RL that tolerates stale answers
Here is the catch. By the time a group of answers is finished and graded, the learner may already have updated the model. Textbook on-policy RL insists on answers from the current weights, which means the learner waits for fresh answers and the engines wait for new weights. That is the stop-and-go loop again, just inside the kernel.
MegaRL instead lets answers be a little stale, under three rules:
- Pin the weights. When an engine starts a group, the group pins the newest published snapshot and writes all four answers with it. Snapshots never change once published: the learner writes new weights into a free slot (there are 12) and never overwrites a snapshot that a running group is still reading.
- Limit the lag. When the learner takes a group, its lag is the learner's version minus the group's version. The group is trained on only if the lag is at most 8. Queued groups that go over are dropped, and groups still generating that can never come back under the limit are cancelled early.
- Correct for the gap. Every answer stores the log-probabilities it was written with. The loss weights each token by the ratio of its probability under the current weights to its probability when it was written, and clips that ratio to [0.8, 1.2], the same clipping PPO uses, so stale answers cannot push the model too far.
The payoff is large. We ran the same kernel twice on two tasks, training layer 13 with 8 groups per update: once requiring lag 0, so every update waits for fresh answers, and once allowing lag 8. Each run had its admission limit and buffers sized for its lag. Allowing lag 8 made each update 11× faster.
In practice the answers are not very stale. In the race, the average trained group was 2.55 updates old, and only 16 groups in the whole run went past the limit.
8. Keeping four tasks fed
With four tasks sharing one learner, a lot of bookkeeping happens on the GPU. Each group moves through a small state machine: generating, graded, then queued or dropped, then trained on or dropped as stale. Graded groups go into lock-free queues in GPU memory, one per task, and their buffers return to a shared free list when the learner is done with them.
- Admission limit. An engine starts a new group for a task only while that task has fewer than 32 groups generating or queued. Producing far more answers than the learner can use would just make them stale.
- Deficit scheduling. Both starting new groups and building each training batch favour the task that has received the least so far, counting what is trained, queued and still generating. That keeps the four tasks balanced without a fixed schedule.
- Back-pressure. When every buffer is in use, engines wait for the learner to free one rather than overwrite data.
The flow chart shows why generation speed matters so much: 81% of graded groups carried no learning signal, because all four answers were right (or all wrong). For every group the learner trained on, the engines wrote about five. The run's built-in audit checks the protocol (every trained group within the lag limit, no snapshot overwritten while in use, and so on) and found 0 violations.
9. The race
Three systems, one H100 80GB, the same model, the same trained layer, the same training prompts, the same GRPO update (16 groups, learning rate 4e-5, gradient clip 1.0, lag limit 8) and the same held-out grader. Each ran for about 26 seconds of training. Setup time (loading weights, building the prompt caches) is excluded for everyone: 4.2 s for MegaRL and 7.6 s for HF PyTorch.
Speed only matters if learning keeps up, so every system's weights were saved during the run and scored afterwards on the 1,600 held-out questions. All three start at 58.6%.
Here is the race as a replay. Both sides run on the clock of their recorded traces: the PyTorch panel shows generation stopping whenever it trains, and the MegaRL panel shows tokens and updates piling up from a single launch.
The takeaway: the win is speed, not a smarter algorithm. All three systems learn to the same accuracy; MegaRL gets there sooner because it generates 11× faster than HF PyTorch and 1.7× faster than a carefully tuned PyTorch loop, from one kernel launch.
10. What we don't claim
- Scale. One H100, a 0.5B-parameter model, answers of at most 64 tokens, multiple-choice tasks, and one seed per system. Bigger models, long reasoning chains and multiple GPUs are untested.
- One trained layer. We train all of layer 13 (14.9M parameters), not the whole model.
- Accuracy is a tie. Final accuracy is within noise across the three systems; the claim is time, not a better final model.
- Setup excluded. All rates count training time only, for every system.
- Separate staleness experiment. The lag 0 against lag 8 comparison is a pair of two-task runs, each configured for its lag, not the race configuration.
- Slightly off-policy. Training on answers up to 8 updates old changes the algorithm a little. The clipped ratio kept it stable here; longer runs may need a tighter limit.
Acknowledgements
Thanks to Flapping Airplanes for the demo that started this, and to Jerry Zhu and Arnold Xu for open-flapping-airplanes, whose open persistent-kernel design MegaRL's scheduler builds on. Thanks to the Qwen team for Qwen2.5-0.5B-Instruct; to the authors of ARC, SciQ and CommonsenseQA; to the DeepSeekMath authors for GRPO; and to Hugging Face for transformers and TRL, whose GRPO implementation we checked our loss against.