November 2026 · 14 min read

Making One GPU Go Brrr: Reinforcement Learning in a Single CUDA Kernel

How we fit the whole RL loop for a language model (writing answers, grading them, training on them and shipping new weights) into one GPU kernel that never exits, and raced it against PyTorch on the same H100.

Videet Mehta · MIT

Table of contents
  1. TL;DR
  2. Goal
  3. RL in one minute
  4. Where the time goes
  5. The inspiration
  6. Scheduling 132 SMs from inside
  7. What gets trained
  8. RL that tolerates stale answers
  9. Keeping four tasks fed
  10. The race
  11. What we don't claim

TL;DR

1. Goal

Reinforcement learning (RL) is how a language model learns from its own attempts: it answers a question, a program checks the answer, and the model is nudged toward the answers that scored well. It is also hard on hardware. The GPU spends most of its time writing answers one token at a time, and the usual training loop keeps interrupting that work to train.

I wanted to see how fast a single GPU could run the whole online RL loop if nothing ever left the GPU: sample answers, grade them, compute the update, publish the new weights, and keep going. Then I wanted to race it against PyTorch on the same GPU, with the same model, prompts and training step, and check that the faster system actually learns as well.

This post tells that story without the gory details: where the time goes in a normal loop, the project that inspired the design, how the kernel schedules its work, the small change to the RL algorithm that made it possible, and the race.

2. RL in one minute

We use GRPO (Group Relative Policy Optimization), a popular recipe for training language models with RL. For each question, the model writes four answers, called a group. A grader gives each answer reward 1 if its final Answer: X is correct and it ends properly, and 0 otherwise. Each answer's advantage is how much better it did than the group's average, divided by the group's spread. The update then makes positive-advantage answers more likely and negative ones less likely.

One questionWhat is 251 + 345?A. 638B. 941C. 596D. 586asked 4 times4 sampled answersrewardadvantage...so the sum is 596. Answer: C1+0.87...about 586. Answer: D0-0.87251 + 345 = 596. Answer: C1+0.87...probably 638. Answer: A0-0.87
Figure 1. One GRPO group (illustrative answers). Two answers are right and two are wrong, so the right ones get advantage +0.87 and the wrong ones −0.87. If all four answers had scored the same, the group would carry no signal and be thrown away.

Our setup is small enough to fit on one GPU and quick to measure. The model is Qwen2.5-0.5B-Instruct, learning four multiple-choice tasks at once: generated arithmetic, ARC science questions, SciQ and CommonsenseQA. Answers are at most 64 new tokens, and each update trains on 16 groups (64 answers). Every result below is scored on 1,600 held-out questions (400 per task) that are never trained on.

Two things decide how fast this goes: how quickly you can produce answers, since generation is most of the work, and how long the generator sits idle while the trainer runs.

3. Where the time goes in a normal RL loop

Launch overhead. A program that runs on the GPU is called a kernel. PyTorch runs a model as a long sequence of small kernels, each launched by the CPU. For a small model writing one token at a time, each kernel finishes in microseconds, and the GPU spends much of its time waiting for the CPU to send the next one. Our straightforward Hugging Face (HF) PyTorch loop made 1,327,001 launches in 26.6 seconds, about 50,000 per second.

PyTorch: the CPU launches every operation as its own kernelHF PyTorch: 1,327,001 launches in 26.6 s; each gap is time the GPU waits for the next oneMegaRL: one launch; its blocks run every task of every job until the run ends
Figure 2. PyTorch issues one small kernel after another from the CPU, and every gap is idle GPU time. MegaRL is a single kernel whose blocks pack generation tasks (blue) and learner tasks (purple) back to back.

Stop-and-go. The usual loop generates for a while, stops generating to train, then resumes. A hand-tuned PyTorch loop fixes most of the launch overhead: it keeps 512 answers rolling, captures decoding in a CUDA graph and uses a static KV cache. That cuts launches to 253,525 and lifts generation to 40.1k tokens/s. But it still stops generating for every update: 179 times in the race, a median of 34 ms each, or 22% of the run.

generatinggeneration stopped to trainlearner computinglearner waitingHF PyTorch5 updatesTuned PyTorch29 updatesMegaRL enginesMegaRL learner43 updates0.0 s0.5 s1.0 s1.5 s2.0 s2.5 s3.0 s
Figure 3. The first three seconds of each run, drawn from the recorded traces. Black marks time when generation was stopped to train. MegaRL's engines (blue) never stop; its learner alternates between computing an update (dark purple) and waiting for its next batch of answers (light purple). Update counts are at the 3-second mark.

So the goal became concrete: no CPU in the loop, and a generator that never stops.

4. The inspiration: a kernel that never exits

The idea came from a demo video by Flapping Airplanes: one GPU kernel that trains a model, serves it and plays Doom, all at the same time. Jerry Zhu (Shopify) and Arnold Xu (Magni) rebuilt it from the video, in the open, as open-flapping-airplanes. It is a persistent megakernel: one launch, and its blocks never exit. They pull training, serving and Doom tasks from queues in GPU memory until the run ends.

open-flapping-airplanes live dashboard: a training-loss curve, text streaming from the model being trained, and Doom, all from one kernel launch
Figure 4. The open-flapping-airplanes dashboard: training loss, live text from the still-training model, and Doom, all from a single kernel launch. Image from the open-flapping-airplanes repository.

Watching it, the thought was simple. RL is just two programs, a generator and a trainer, that share a GPU and pass data back and forth. If one kernel can run a trainer, a server and Doom side by side, it can run the two halves of RL side by side, and the generator never has to stop. MegaRL's scheduler starts from open-flapping-airplanes' design and adds what RL needs: model versions, queues of graded answers, and rules for how stale an answer may be.

5. Scheduling 132 SMs from inside the kernel

An H100 has 132 streaming multiprocessors (SMs), the GPU's independent cores. MegaRL launches exactly two blocks of 256 threads on each SM, 264 blocks in all, once. Every block then loops: grab a task, run it, report that it is done, repeat, until the run ends.

Jobs, phases and tickets

The work is organized as jobs: four generation engines and one learner. A job advances in steps. One step of an engine produces the next token for each of its 128 answers; one step of the learner is one training update. A step is a fixed list of phases (normalize, project to queries/keys/values, attention, MLP, and so on, layer by layer), and each phase is split into many small tasks that can run in parallel on different SMs.

Every task gets a number, its ticket. A block takes the next ticket with a single atomic add, runs that task, then bumps the phase's completion counter. When a phase's counter is full, the next phase opens. Because a task only ever depends on tasks with lower tickets, whoever holds the lowest unfinished ticket can always make progress, so the scheduler cannot deadlock.

one step of a generation engine = the next token for each of its 128 answersQKV01234/4 doneattention4567893/6 doneMLP101112130/4 doneLM head1415160/3 donesample170/1 doneblock on SM 7took ticket 7block on SM 41took ticket 8block on SM 88took ticket 9ticket counter: 10ticket 10 (MLP) is handedout once attention is 6/6A phase opens once every task it depends on has bumped its completion counter.
Figure 5. Tickets and completion counters, simplified (a real step has about 200 phases, and a phase can have hundreds of tasks). Three blocks on different SMs are working on attention tasks 7 to 9. The MLP phase opens, and ticket 10 is handed out, when all six attention tasks have reported in.

Who runs what

Each block has a home job. The first block on every SM belongs to a generation engine, and each engine owns a contiguous range of SMs. The second block on every other SM belongs to the learner; on the rest it is another generation block. That gives 198 generation blocks and 66 learner blocks.

engine 1engine 2engine 3engine 4learner132 SMs, 2 blocks each (top and bottom half). 198 blocks generate, 66 learn.
Figure 6. The H100's 132 SMs, each drawn as two halves for its two blocks. Blues are the four generation engines; purple is the learner, which is spread evenly over the chip.

One rule keeps every block busy: a block never takes a ticket it cannot run yet. While the learner waits for a complete batch of answers, its blocks run generation tasks instead of spinning. The matrix multiplies inside the tasks use Hopper's warpgroup tensor-core instructions (wgmma), so each task is fast, not just well scheduled.

6. What gets trained, and why the replay is exact

We train all of layer 13, one of the model's 24 transformer layers: its attention and MLP weights, 14,946,304 parameters in all.

The trained layer is used from the last prompt token onward, while the prompt itself always goes through the original weights. So each question's KV cache is computed once at setup and reused by every version of the model. While generating, the engines save the hidden state entering layer 13 for every token. The learner starts from those saved states (layers 0 to 12 never change), replays layers 13 to 23 with exactly the same arithmetic the engines used, and backpropagates from layer 23 down to 13. Because the replay matches, the learner's log-probabilities equal the generation-time ones bit for bit when the weights have not changed, so any difference it sees is real policy change, not numerical noise.

Each update clips the gradient to norm 1.0, applies AdamW on full-precision master weights, and publishes a 29.9 MB bf16 snapshot of the new layer for the engines to use. In the race, an update took about 65 ms of computing plus about 37 ms of waiting for a complete batch, time the learner's blocks spent helping generate.

7. RL that tolerates stale answers

Here is the catch. By the time a group of answers is finished and graded, the learner may already have updated the model. Textbook on-policy RL insists on answers from the current weights, which means the learner waits for fresh answers and the engines wait for new weights. That is the stop-and-go loop again, just inside the kernel.

MegaRL instead lets answers be a little stale, under three rules:

  1. Pin the weights. When an engine starts a group, the group pins the newest published snapshot and writes all four answers with it. Snapshots never change once published: the learner writes new weights into a free slot (there are 12) and never overwrites a snapshot that a running group is still reading.
  2. Limit the lag. When the learner takes a group, its lag is the learner's version minus the group's version. The group is trained on only if the lag is at most 8. Queued groups that go over are dropped, and groups still generating that can never come back under the limit are cancelled early.
  3. Correct for the gap. Every answer stores the log-probabilities it was written with. The loss weights each token by the ratio of its probability under the current weights to its probability when it was written, and clips that ratio to [0.8, 1.2], the same clipping PPO uses, so stale answers cannot push the model too far.
learnerv0v1v2v3v4v5v6v7v8v9v10v11v12pinned v3, trained at v5: 2 behindpinned v4, trained at v9: 5 behindpinned v2, reached at v11: 9 behind, droppedgenerating with its pinned weightswaiting in the queue
Figure 7. Version pinning and the lag window. Each group pins the newest weights when it starts (open circle). The learner trains on it if it is at most 8 versions behind; the bottom group waited too long and is dropped.

The payoff is large. We ran the same kernel twice on two tasks, training layer 13 with 8 groups per update: once requiring lag 0, so every update waits for fresh answers, and once allowing lag 8. Each run had its admission limit and buffers sized for its lag. Allowing lag 8 made each update 11× faster.

Time per updatelag 0: wait for fresh answers618 mslag 8: MegaRL default57 msTokens generated per secondlag 0: wait for fresh answers2.8klag 8: MegaRL default34.0k
Figure 8. The same kernel with lag 0 (20 updates) and lag 8 (100 updates), two tasks, layer 13. With lag 0 the engines sit idle waiting for each update, so generation drops too.

In practice the answers are not very stale. In the race, the average trained group was 2.55 updates old, and only 16 groups in the whole run went past the limit.

31095411,52427083310421151176947518updates behind when trained (4,000 trained groups, mean 2.55, limit 8)
Figure 9. How many updates behind each trained group was in the race run. Most groups are one to three updates old.

8. Keeping four tasks fed

With four tasks sharing one learner, a lot of bookkeeping happens on the GPU. Each group moves through a small state machine: generating, graded, then queued or dropped, then trained on or dropped as stale. Graded groups go into lock-free queues in GPU memory, one per task, and their buffers return to a shared free list when the learner is done with them.

groups started20,825answered and scored20,711all 4 rewards equal: no signal, dropped16,685 (81%)usable, queued4,026trained on4,000too stale16+ 10 still queued at the end114 still generating at the end
Figure 10. What happened to every group in the race run. Most groups are thrown away because all four answers got the same reward.

The flow chart shows why generation speed matters so much: 81% of graded groups carried no learning signal, because all four answers were right (or all wrong). For every group the learner trained on, the engines wrote about five. The run's built-in audit checks the protocol (every trained group within the lag limit, no snapshot overwritten while in use, and so on) and found 0 violations.

9. The race

Three systems, one H100 80GB, the same model, the same trained layer, the same training prompts, the same GRPO update (16 groups, learning rate 4e-5, gradient clip 1.0, lag limit 8) and the same held-out grader. Each ran for about 26 seconds of training. Setup time (loading weights, building the prompt caches) is excluded for everyone: 4.2 s for MegaRL and 7.6 s for HF PyTorch.

67.7ktokens/s generated by MegaRL, 11× HF PyTorch and 1.7× tuned PyTorch
156groups trained per second, against 108 (tuned) and 32 (HF)
1kernel launch for the whole run, against 1,327,001 for HF PyTorch
tokens generated per secondkernel launchesupdatesPyTorchHF transformers6.1k1,327,00154PyTorch, hand-tunedCUDA graphs, static KV cache40.1k253,525179MegaRLone persistent kernel67.7k1250One H100, same model, prompts and training step; about 26 s of training each, set-up excluded.
Figure 11. Generation speed, kernel launches and updates completed for the three systems. MegaRL finished its 250 updates in 25.6 s; the PyTorch loops completed 179 and 54 in the same window.

Speed only matters if learning keeps up, so every system's weights were saved during the run and scored afterwards on the 1,600 held-out questions. All three start at 58.6%.

55%60%65%70%75%80%0 s5 s10 s15 s20 s25 sTuned PyTorch 75.6%MegaRL 74.8%HF PyTorch 73.6%70%seconds of training
Figure 12. Held-out accuracy (mean over the four tasks) against seconds of training. MegaRL passes 70% after 1.5 s; HF PyTorch needs 8.6 s. The tuned loop was scored only every 25 updates; its first score, at 2.5 s, was already above 70%, so we can't say which of it and MegaRL got there first. At the end the three are within two points, which is within noise for 1,600 questions (one standard error is about 1.1 points).

Here is the race as a replay. Both sides run on the clock of their recorded traces: the PyTorch panel shows generation stopping whenever it trains, and the MegaRL panel shows tokens and updates piling up from a single launch.

The race replay at 8 seconds: PyTorch on the left, MegaRL on the right, with tokens per second, updates and a held-out accuracy chart
Figure 13. The race replay, built from the recorded runs (HF PyTorch against MegaRL; the tuned loop appears on the final card). Press Play; the controls inside let you pause, seek and change speed.

The takeaway: the win is speed, not a smarter algorithm. All three systems learn to the same accuracy; MegaRL gets there sooner because it generates 11× faster than HF PyTorch and 1.7× faster than a carefully tuned PyTorch loop, from one kernel launch.

10. What we don't claim

Acknowledgements

Thanks to Flapping Airplanes for the demo that started this, and to Jerry Zhu and Arnold Xu for open-flapping-airplanes, whose open persistent-kernel design MegaRL's scheduler builds on. Thanks to the Qwen team for Qwen2.5-0.5B-Instruct; to the authors of ARC, SciQ and CommonsenseQA; to the DeepSeekMath authors for GRPO; and to Hugging Face for transformers and TRL, whose GRPO implementation we checked our loss against.