A new public leaderboard called LoRA Speedrun ranks fine-tuning techniques by raw wall-clock time. The results expose which tricks actually matter, and which are just marketing.
A public wall-clock leaderboard for fine-tuning has been overdue for years. Every paper claims speedups. Every framework claims efficiency. And almost nobody reproduces the numbers on the same hardware, same seed, same eval budget. That's the gap LoRA Speedrun is trying to close, and after landing on GitHub in mid-July 2026 with a self-described tagline of "modded-nanogpt for fine-tuning", it's the clearest attempt yet to force LoRA methods into head-to-head, reproducible races.
The premise is stupidly simple. Fix the model, fix the dataset, fix the target accuracy. Then race. Whoever hits the target first, wins. No asterisks about "equivalent quality." No hand-wavy FLOP counts. Just a clock.
LoRA Speedrun is a public wall-clock leaderboard for parameter-efficient fine-tuning. Contributors submit a training script and a reproducible run that clears a fixed accuracy target on a frozen task. The fastest wall-clock time wins the slot, and every record is independently re-run three times with fresh seeds on identical hardware before it counts.
Photo by Domaintechnik on Unsplash
And that constraint changes everything. When you can't cheat by training longer or evaluating on a friendlier split, most of the paper-famous "efficient fine-tuning" methods have to prove themselves against the clock.
The project takes explicit inspiration from Keller Jordan's modded-nanogpt speedrun, which turned pretraining into a competitive sport and produced real algorithmic wins like the Muon optimizer. LoRA Speedrun aims at a much bigger audience: the engineers who fine-tune models on their own data and don't much care whether your method is "theoretically optimal" if it takes 3x longer on their single card.
The rules, per the current README, are pretty tight:
Submissions land as PRs. CI validates them, an automated security review reads the diff for network exfiltration or harness tampering, and a maintainer triggers verification with a /verify comment that re-runs the submission in a network-blocked Modal sandbox on the spec hardware. The harness also audits adapter parameter counts and re-verifies model and data hashes to catch tampering.
The whole thing works because someone is willing to press "go" on other people's code in a sealed sandbox and publish the timer honestly. That's the boring, unglamorous piece that makes the leaderboard trustworthy.
One nice detail: because verification runs on Modal's free monthly compute credits, attempting a run and re-verifying any record on the board are both free for anyone who signs up.
The board is young. As of the July 20, 2026 snapshot, Track 1 (GSM8K on Qwen2.5-1.5B, L40S) has two entries:
| # | Date | Author | Wall-Clock | GSM8K EM | Technique |
|---|---|---|---|---|---|
| 0 | 2026-07-18 | maintainer | 11m 57s | 59.4% | Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR |
| 1 | 2026-07-18 | maintainer | 6m 05s | 61.1% | Sequence packing + completion-only loss masking, 2 epochs, same LoRA config as #0 |
Track 2 (SQuAD v1.1 on SmolLM2-1.7B, L40S) currently has one entry:
| # | Date | Author | Wall-Clock | SQuAD EM | Technique |
|---|---|---|---|---|---|
| 0 | 2026-07-20 | maintainer | 11m 08s | 77.5% | Baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss |
Both records so far belong to the maintainer, which is normal for a leaderboard's first week — the interesting phase starts when outside submissions land.
The record #1 entry on Track 1 uses sequence packing plus completion-only loss masking, and it comes in at roughly half the baseline's wall-clock (6m 05s versus 11m 57s) while landing at higher accuracy. That's a clean example of what the leaderboard is designed to expose: two well-known techniques, combined, produce a large speedup that would be easy to overlook in a paper because the accuracy delta on its own isn't headline-worthy.
Only one hardware SKU, only one adapter-parameter budget, only two tasks. That's the point. Comparability is the product. The README explicitly rejects a consumer-GPU track because rented consumer cards vary too much host-to-host to yield stable numbers; the L40S on Modal is the same AD102 silicon as the RTX 4090, so consumer-GPU kernel tricks should still transfer to the leaderboard.
The README lists techniques still open for the taking: 1-epoch aggressive-LR schedules, block-diagonal or varlen packing, QLoRA NF4 vs bf16 tradeoffs, rsLoRA, DoRA, PiSSA init, LoRA+ (asymmetric learning rate for A and B), NEFTune, curriculum ordering, MLP-only vs attention-only placement, torch.compile, Unsloth kernels, Liger kernels, fused cross-entropy. Any of those, applied correctly, plausibly beats 6m 05s.
The honest read is that the LoRA ecosystem has been sleepwalking on head-to-head reproducibility for two years. Rank techniques by paper impact and DoRA, LoRA-FA, and various "efficient" variants dominate the conversation. But when you rank by wall-clock on identical hardware to a fixed accuracy target, the winner is whoever ships the best combination of algorithmic and systems tricks — and the leaderboard is where those combinations get to fight in public.
That's not a knock on the algorithm work. It's a knock on how we've been benchmarking. Reporting steps-per-second on your own cluster with your own eval protocol is essentially a marketing exercise. Reporting time-to-target on a shared sandbox with 3-seed verification is a science experiment.
The modded-nanogpt speedrun shows the pattern at pretraining scale. Its GPT-2 (124M-class) target — 3.28 cross-entropy loss on FineWeb, chosen because Karpathy's llm.c reference run hit that same loss in about 45 minutes — has been driven down to under 90 seconds on 8xH100 through many incremental submissions and produced techniques like Muon along the way. Most of those wins came from architecture tweaks and optimizer changes that never had a dedicated paper.
Photo by James Harrison on Unsplash
LoRA Speedrun is betting the same phenomenon exists for fine-tuning. The first record already suggests it might.
Beyond the top-line rankings, a few design choices are worth calling out.
The baseline is deliberately soft. Plain LoRA r=16 on all linear layers, cosine LR, no tricks. That leaves plenty of room for cheap wins from anyone who applies the standard bag of throughput techniques — packing, masking, aggressive LR, kernel swaps — and it's already produced a ~2x speedup within the first day.
Adapter parameter cap keeps the race honest. The ≤ 30M trainable-parameter budget prevents a "who can afford more adapter capacity" arms race and forces submissions to compete on algorithm and systems quality rather than raw parameter count.
Two tracks with different model families make transfer testable. A trick that only wins on Qwen2.5-1.5B math might just be exploiting math-friendly pretraining. A trick that wins on both Track 1 and Track 2, on different families and different task types, is the kind of technique worth actually adopting.
Verification is sandboxed and network-blocked. The submission harness runs strangers' code in a Modal sandbox with no outbound network and no secrets. That's why the maintainer can afford to keep re-verifying: the personal-machine risk that would kill this format for most people is absorbed by Modal's isolation.
If you fine-tune models regularly, the leaderboard has some immediate takeaways.
First, the record #1 result is a reminder that sequence packing plus completion-only loss masking is nearly free money on instruction-style data. If you're still training on padded, full-sequence-loss batches, that's often a 30-50% wall-clock miss on this kind of workload.
Second, don't pick your fine-tuning method based on paper velocity. Pick it based on reproducible wall-clock plus a downstream eval that matches your task. Some methods might give better final quality on some workloads, but if training speed matters, watch what actually posts to the board versus what's just been benchmarked in isolation.
Third, if you're a framework author, submit. Companies including HuggingFace have shipped official Unsloth integration paths for TRL, and there's a natural incentive to prove those kernel wins hold up under leaderboard conditions. The leaderboard is the fastest way to prove your claims aren't cherry-picked. And if your framework loses, you learn where the hot path actually is.
The leaderboard is young. Numbers will move. Submissions will invalidate what's above. That's the point.
You don't need to own one. Verification runs on a Modal L40S sandbox, and Modal's free monthly compute credits are enough to cover full timing runs — so anyone with a Modal account can compete and re-verify records. You can iterate locally on any 24 GB+ card, but local times aren't official; the leaderboard clock is the Modal L40S.
Quality is fixed, not maximized. Every submission must clear a hard exact-match target (≥ 57% on GSM8K for Track 1, ≥ 75.5% on SQuAD v1.1 for Track 2) before the clock stops. This prevents cheating by training a worse model faster, but it also means the board doesn't reward methods that could push accuracy higher with more time.
Papers with Code aggregates self-reported numbers across different hardware, seeds, and eval protocols. LoRA Speedrun re-runs every submission 3 times with fresh seeds on the same Modal L40S sandbox with pinned dependencies, and only records the mean if all three seeds clear the target. Far fewer entries, but the ones on the board are actually comparable.
The rules cap submissions at 30M trainable parameters and require adapter-only training, so full fine-tunes are structurally out of scope. The point of the leaderboard is to compare parameter-efficient methods against each other on identical hardware, not to establish a full fine-tune ceiling.
Not on this board. The README is explicit: one L40S, adapter-only, ≤ 30M trainable params — 'That's the point.' Track 2 (SmolLM2-1.7B on SQuAD) already gives a second model family and task type for testing whether techniques transfer, and more tracks in that mold are likely before any move to bigger models or multi-GPU setups.
