Pretraining a 0.5B model on one rented GPU

I trained a 489M-parameter Llama-style model from scratch on a single rented RTX 5090. The run is finished: three phases, 4.19 billion tokens, $13.70 of GPU time. This post covers what I built, how it scores, and two things I tried that did not work.

The final weights are on the Hub as osjayaprakash/llm-0.5b-fineweb-edu, under MIT. It loads as a plain LlamaForCausalLM, so nothing custom is needed to run it.

The setup

The model is a standard decoder-only transformer, written in plain PyTorch:

Parameters

489.3M (448.3M excluding embeddings)

Layers and width

24 layers, d_model 1280

Attention

Grouped-query, 20 query heads sharing 4 key/value heads

Feed-forward

SwiGLU, d_ff 3840

Normalization and positions

Pre-norm RMSNorm, RoPE

Vocabulary

32,000-token byte-level BPE, trained on the same corpus

Context

2048 tokens

Batch

524,288 tokens per step

The training data is 2.03B tokens of FineWeb-Edu, with 50M held out for validation. The GPU was a 32 GB RTX 5090 rented on Vast.ai at $0.479 an hour. It trained at 48,000 tokens a second, about 85% of the card’s theoretical peak.

Results

Each doubling of the tokens paid, and each paid a little less than the one before. The first took validation loss down 0.18 and average accuracy up 2 points; the second, 0.14 and 1.2 points. Neither curve has bent toward a floor. At 4.2B tokens the model has seen about 8.5 tokens per parameter, still under half the Chinchilla-optimal ratio. Tokens were the limit from start to finish. I stopped because the budget ran out, not because the model stopped learning.

Training was stable throughout: across all 8,000 steps there were no loss spikes and no non-finite gradients, and gradient clipping stopped triggering after step 540.

Against public models

I ran every model through the same lm-evaluation-harness tasks on the same machine. Before evaluating my checkpoints, I converted them to the Hugging Face Llama format and confirmed the converted model gives the same outputs as the original.

At a matched 2.1B tokens, my model scored 8.5 points above Pythia-410M. Finishing at 4.19B puts it 9.7 points above that checkpoint and 4.8 points behind the fully trained Pythia-410M, which saw 72 times as much data.

SmolLM2-360M beats Qwen2.5-0.5B with fewer parameters and less than a quarter of the tokens. That gap comes from how its training data was chosen and mixed. I haven’t worked on data selection at all yet, so it’s the most promising next step once tokens stop being the bottleneck.

On MMLU (5-shot) the model finished at 0.272, up from 0.254 at half the tokens. Chance is 0.25 on a four-way multiple choice, so this is the first sign of that kind of knowledge appearing, and not much more than a sign: social sciences carries it at 0.320 while humanities is still at chance. I leave MMLU out of the average.

Restarting the learning-rate schedule has a cost

Each phase used a cosine schedule: warm the learning rate up, then decay it to a tenth of its peak. To continue after phase 1, I warmed the learning rate back up to 2e-4 and decayed it again.

Raising the learning rate on a model that had just been annealed undid part of that annealing. Validation loss rose from 3.323 to 3.383 and took 600 steps to get back below 3.323, about a third of phase 2. The shaded areas in the chart are that cost.

Phase 3 repeated it and showed what governs the size of the bill. It rose from 3.139 to 3.193, then needed 1,100 steps to recover, nearly twice phase 2’s 600. The restart itself wasn’t worse; phase 3’s cosine simply runs over 4,000 steps instead of 2,000, so it holds the learning rate near its peak for twice as long. The cost tracks how long the schedule stays hot, not the act of restarting.

Both phases finished well ahead, so both restarts were worth doing. But together they spent 1,700 of the run’s 8,000 steps, 21% of the budget, re-earning ground already taken. A warmup-stable-decay schedule avoids this entirely: hold the learning rate constant and decay it only at the very end. It’s the better choice whenever the total training budget isn’t fixed in advance, which for me it usually isn’t.

What moved and what didn’t

LAMBADA, which asks the model to predict the last word of a passage, rose elevenfold from 0.021 to 0.230, making it the most useful single sign of progress. SciQ nearly doubled. HellaSwag sat nearly still for two phases and then finally began to move in phase 3, from 0.298 to 0.326.

ARC-Challenge ended at 0.265 and WinoGrande at 0.502, both at or near chance after the whole run, and Pythia’s checkpoints at the same token counts behave the same way. At this scale those two cost evaluation time without informing a single decision. I’d drop them from the tracking suite and watch validation perplexity and LAMBADA instead.

Averaging the weights didn’t help

Phase 3 also kept an exponential moving average of the weights, on the usual reasoning that averaging away the noise the last steps leave in each parameter is worth a few tenths of a point. It wasn’t. The averaged weights scored 0.443 against the final checkpoint’s 0.445, and were slightly worse on validation perplexity too, 42.5 against 42.1. That’s within noise, but it is certainly not a gain.

The reason makes sense in hindsight. Averaging helps when training ends while the learning rate is still high and the weights are still being jostled around. This run ends at 1e-5 after a full cosine decay, so the last thousand steps are already taking tiny, quiet steps. The decay had done the averaging already. The two techniques are substitutes, not complements.

So: if you can afford to decay the learning rate properly, do that and skip the averaging. Keep it for runs that end while the rate is still high, which is what happens with a constant-rate schedule, an early stop, or a budget that runs out mid-decay. Finding this out cost one gigabyte of disk and no measurable training time, which is a fair price for knowing.

Training health

I also logged per-layer gradient norms, activation sizes, and prediction entropy throughout. None of them needed action this time, but each would have caught a problem long before it showed up in the loss curve.

The first layer carries the largest gradients throughout, four to six times those of the early-middle layers, and the second half of the network runs higher than the first. All 24 layers shrink together, and each learning-rate restart shows up as a faint band at steps 2000 and 4000. No layer ever drifted away from the rest across the whole run. A layer whose gradients grew or collapsed on its own would be the first sign of an unstable run, and nothing here ever needed me to intervene.

The residual stream tells a different story: it’s banded by depth, not by time. Each layer adds its output to a running sum that only gets normalised on the way into the next block, so the stream grows steadily from layer 0 at the bottom to layer 23 at the top. That gradient is the signature of a healthy pre-norm network. What you don’t want is a single row suddenly brightening, which is what a layer heading for numerical trouble looks like well before the loss notices.

Prediction entropy drops steeply during warmup, from 7.44 nats to about 3.5 by the end of phase 1, and then stops falling. It spends phases 2 and 3 wandering around 3 nats, bouncing half a nat either way because I measure it on a single batch. That plateau is worth noticing: the model kept getting better at predicting the right token long after it stopped getting more confident on average. Loss and confidence are not the same thing, and only one of them was still improving.

The update sizes shrink as the learning rate decays and step back up at each restart, which is the loss curve’s story told from the optimizer’s side. None of this needed action. That’s rather the point of logging it: it’s cheap, and the run where one of these goes wrong is the run where you want it already there.

Cost

What I’d do next

The run is done, and it never hit a wall I could fix with engineering. Ranked by what would actually pay:

Change the data mix, don’t just add more of it. The cheap token gains are largely spent, and the corpus is the binding constraint now. The benchmarks that stayed flat are flat because FineWeb-Edu contains none of what they test: no code, little reference text, nothing requiring multi-step reasoning. SmolLM2-360M beating Qwen2.5-0.5B on a quarter of the tokens is the evidence that mixture outweighs volume from here.

If you do add tokens, use one warmup-stable-decay schedule. Another doubling would still pay, since the curve is straight to the end. Running it as a single schedule recovers the fifth of the budget my two restarts cost.

Skip the weight averaging if you can decay properly, per the measurement above.

Drop ARC-Challenge and WinoGrande from the tracking suite at this scale. Both sat at chance for the whole run.

What I wouldn’t bother with at this budget: architecture changes, a different optimizer, or a hyperparameter sweep. Nothing in the diagnostics ever suggested the model was the limiting factor. No instability, no dead layers, no gradient pathology, 85% of the card’s peak throughput from the first step to the last. It was the data all along.

Getting the model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")
tok = AutoTokenizer.from_pretrained("osjayaprakash/llm-0.5b-fineweb-edu")

It’s a base model: no instruction tuning, no chat template, no alignment. It continues text and nothing else, and at this size it produces fluent, plausible, frequently wrong prose. Ask it about mitochondria and it will tell you they are the powerhouse of the cell, three times, before moving on. Treat it as a reference point for what a small budget buys rather than as something to build on. The model card carries the full numbers and limitations.