# bf16 memory test and GPU choice (2026-10-06, estimates, nothing was run)

**No GPU job is started without Kral's go.** Script: `train/hf_train_bf16.py` (`--memory-test --sweep 16000,32000,48000` is the memory test, no data needed, 3 optimizer steps per length, stops at the first OOM, uploads the result as `memtest_*.json` to the output repo).

## Model (from config.json of Qwen3.8-27B)
64 layers (48 linear attention, 16 full attention), hidden 5120, MLP 17408, vocab 248320, 27.8 B parameters. LoRA rank 16 on q/k/v/o, gate/up/down, in_proj_qkv, in_proj_z, out_proj: **107 M trainable parameters**.

## Memory by component at 48k tokens (GB; the estimate, not a measurement)
| component | GB | how |
|---|---|---|
| weights bf16 | 51.8 | 27.8 B x 2 bytes |
| LoRA weights + grads + 8-bit Adam | 1.0 | 107 M x 10 bytes |
| layer inputs for the backward pass (checkpoints) | 29.3 on the GPU, about 0 with the Unsloth CPU offload | 48k x 5120 x 2 bytes x 64 layers; the offload needs 29 GB of host RAM (all flavors have 142 GB or more) |
| recompute peak of one layer | 9.2 | MLP tensors 48k x 17408 x 2 bytes x 4 plus 3 GB for attention (assumption) |
| logits and loss | 3 chunked, 67 with full logits | full logits: 48k x 248320 x (bf16 + fp32 upcast + grad); only a chunked or fused cross entropy is realistic |

## Fit by GPU (92 % of the card counted as usable)
| seq | checkpoints | loss | estimated peak GB | 1x A100 80 GB | 1x RTX PRO 6000 96 GB | 1x H200 141 GB | 2x H200 282 GB (needs model parallel) |
|---|---|---|---|---|---|---|---|
| 16k | CPU offload (Unsloth) | chunked | 61 | fits | fits | fits | fits |
| 16k | CPU offload (Unsloth) | full logits | 80 | no | fits | fits | fits |
| 16k | on the GPU | chunked | 71 | fits | fits | fits | fits |
| 16k | on the GPU | full logits | 90 | no | tight | fits | fits |
| 32k | CPU offload (Unsloth) | chunked | 63 | fits | fits | fits | fits |
| 32k | CPU offload (Unsloth) | full logits | 104 | no | no | fits | fits |
| 32k | on the GPU | chunked | 82 | no | fits | fits | fits |
| 32k | on the GPU | full logits | 124 | no | no | fits | fits |
| 48k | CPU offload (Unsloth) | chunked | 65 | fits | fits | fits | fits |
| 48k | CPU offload (Unsloth) | full logits | 129 | no | no | fits | fits |
| 48k | on the GPU | chunked | 94 | no | tight | fits | fits |
| 48k | on the GPU | full logits | 158 | no | no | no | fits |

Reading:
- **The loss is the main risk, not the weights.** With full logits only the H200 (141 GB, with the CPU offload) fits 32k and 48k; the A100 and the RTX PRO 6000 do not. The run needs a fused or chunked cross entropy
  (Unsloth has one for the architectures it patches; whether it covers `qwen3_5` is not known). The memory test shows it at once: if `seq_len 16000` already fails on an H200, the loss is the cause.
  Fallback (not built yet, small): compute the hidden states, then the loss over the 32 % labeled positions only, in chunks of 4k.
- **A100 80 GB (2.50 USD/hour) is only possible with the CPU offload and a chunked loss** (about 65 GB estimated at 48k: no margin). H200 141 GB (5 USD/hour) fits with margin;
  RTX PRO 6000 96 GB (2.75 USD/hour) fits with the offload and the chunked loss (est. 65 GB).
- Samples over 32k are a minority (p90 39560 tokens in the current data, p50 24.8k): if 48k does not fit, 32k would drop 15 of 52 samples (29 %; 10 of them DDLS), so test 48k first.
- Multi-GPU (`a100x4`, `h200x2`): possible only with model parallelism (`device_map`), one card works at a time, Unsloth multi-GPU is limited; not recommended before the single card test.

## Proposed order
1. Memory test on **h200** (about 30 minutes, 2.5 USD): sweep 16000,32000,48000 with the offload. If 48k fits with margin: stay on H200 or try rtx-pro-6000 for the real run.
2. If the loss is the problem: build the chunked loss (about 2 hours of work), repeat the test.
3. Real run after the stage 1 : stage 2 ratio decision (below).

## Time and cost (estimate from the nf4 run: 221 tokens/s on an A100 at 6.75k tokens per document; bf16 is faster, H200 about 2 to 2.5 times an A100)
Tokens of the proposed run (1 epoch stage 1 and 3 epochs of the 47 stage 2 samples as built today: 2.53 M + 3 x 1.28 M = 6.4 M tokens): A100 about 5.9 h (about 15 USD), H200 about 2.7 h (about 14 USD). The real number comes from the memory test (`step_seconds`).