Stage 1 step 1: prepare.py, real corpus numbers (SAP-samples/abap-cheat-sheets)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014aUaQeLnwbb1zTpN7kHeat
2026-10-03 21:58:04 +02:00
parent f0593933f9
commit e083c9ca13
6 changed files with 593 additions and 5 deletions
--- a/train/STATE.md
+++ b/train/STATE.md
@@ -15,6 +15,11 @@ Task: `docs/stage1-training-task.md`. Settings and weights: `train/README.md`. U
 - Runner: `train/baseline.py --label <label>` (one task at a time, results `runs/stage1/<label>.json`,
  run directories `runs/stage1/<label>/`).

+- Step 1 done (2026-10-03 22:00): `train/prepare.py`, report `train/data/report.md`. Corpus replaced by
+  SAP-samples/abap-cheat-sheets (Apache-2.0): 370 docs, 2.81M real tokens. 45 docs (840k tokens, 30 %) are longer
+  than 16384 and removed. Train 309 docs / 1.89M tokens, valid 16 docs / 80k tokens (split by family, seed
+  20261003), 618 iterations for 2 epochs. `test.jsonl` = copy of valid (needed by `mlx_lm.lora --test`).
+
 ## Running (detached)

 - MLX server PID 59352, log `runs/stage1/server.log`.
@@ -27,8 +32,7 @@ Task: `docs/stage1-training-task.md`. Settings and weights: `train/README.md`. U

 1. B2: read the last lines of `runs/stage1/baseline.log`; summary from `runs/stage1/baseline.json`
   (`t01_test_budget40` holds the T01 test result). Commit.
-2. Step 1: `train/prepare.py` (real token counts with the base model tokenizer, length filter 16384,
-   95/5 split by document, `train/data/train.jsonl` and `valid.jsonl`, report). A4H is not needed.
+2. Decisions for Kral on the data (see report): the 45 removed docs, DOC share, valid size.
 3. Step 2, second part: valid loss of the base model (`mlx_lm.lora --test` without adapter; check the
   options with `--help` first). Add it to `runs/stage1/baseline.json`.
 4. Step 3 (training): Kral stops A4H; stop the MLX server; no other model loaded. Short test of 20
--- a/train/data/report.md
+++ b/train/data/report.md
@@ -0,0 +1,98 @@
+# Stage 1 data report
+
+Corpus: `/Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl` (source abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md).
+Tokenizer: `/Users/erhankeseli/models/Qwen3.8-27B-4bit`. Seed 20261003. max_seq_length 16384.
+Split by family (release versions and parts of one document stay together): 180 families.
+
+| | docs | tokens |
+|---|---|---|
+| corpus | 370 | 2806501 (estimate chars/4: 2636112, ratio 1.065) |
+| removed (> 16384) | 45 | 840136 |
+| train | 309 | 1886475 |
+| valid (= test file) | 16 | 79890 |
+
+Iterations for 2 epochs (batch 1): 618.
+
+## Token distribution per document (real tokenizer)
+
+| set | min | p50 | p90 | p99 | max |
+|---|---|---|---|---|---|
+| all | 49 | 5195 | 16923 | 21483 | 31552 |
+| kept | 49 | 4027 | 14998 | 16217 | 16273 |
+| train | 49 | 4403 | 14998 | 16217 | 16273 |
+| valid | 130 | 3392 | 16027 | 16129 | 16129 |
+
+| tokens per document up to | docs |
+|---|---|
+| 512 | 48 |
+| 1024 | 25 |
+| 2048 | 40 |
+| 4096 | 51 |
+| 8192 | 53 |
+| 16384 | 108 |
+| more | 45 |
+
+## By object type (kept documents)
+
+| types | docs | tokens |
+|---|---|---|
+| DOC | 106 | 1012152 |
+| CLAS | 147 | 827969 |
+| PROG | 13 | 55365 |
+| mixed(4) | 6 | 28405 |
+| DDLS | 27 | 20614 |
+| mixed(3) | 2 | 6667 |
+| DDLX | 6 | 5003 |
+| CLAS+DDLS | 1 | 4563 |
+| FUGR | 2 | 2026 |
+| INTF | 7 | 1792 |
+| BDEF | 6 | 1564 |
+| DCLS | 2 | 245 |
+
+## Removed documents
+
+- ZCL_DEMO_ABAP_OBJECTS (CLAS): 17042 tokens
+- ZCL_DEMO_ABAP_STRUCTURES (CLAS): 16628 tokens
+- 32_Performance_Notes__P03 (DOC): 16742 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 19117 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 18883 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS): 19801 tokens
+- 23_Date_and_Time (DOC): 20347 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS): 17477 tokens
+- 07_String_Processing__P02 (DOC): 17916 tokens
+- ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS): 17368 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS): 31552 tokens
+- ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS): 19499 tokens
+- 04_ABAP_Object_Orientation__P06 (DOC): 16694 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS): 18368 tokens
+- 06_Dynamic_Programming__P07 (DOC): 23994 tokens
+- 28_Regular_Expressions (DOC): 17900 tokens
+- 06_Dynamic_Programming__P06 (DOC): 16923 tokens
+- ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS): 19769 tokens
+- 05_Constructor_Expressions__P01 (DOC): 17349 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS): 17237 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS): 18069 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 17923 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16924 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16770 tokens
+- 14_ABAP_Unit_Tests__P03 (DOC): 18549 tokens
+- ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS): 22096 tokens
+- 22_Released_ABAP_Classes__P01 (DOC): 19022 tokens
+- 21_XML_JSON__P01 (DOC): 16625 tokens
+- 29_Numeric_Operations__P01 (DOC): 17805 tokens
+- 03_ABAP_SQL__P01 (DOC): 17305 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS): 18308 tokens
+- 14_ABAP_Unit_Tests__P04 (DOC): 17633 tokens
+- 24_Builtin_Functions (DOC): 16851 tokens
+- ZCL_DEMO_ABAP_NUMERIC_OP (CLAS): 19740 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS): 17355 tokens
+- ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS): 20446 tokens
+- ZCL_DEMO_ABAP_STRING_PROC (CLAS): 20311 tokens
+- ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS): 16394 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS): 21483 tokens
+- ZCL_DEMO_ABAP_DATE_TIME (CLAS): 18634 tokens
+- ZCL_DEMO_ABAP_XML_JSON (CLAS): 16825 tokens
+- ZCL_DEMO_ABAP_SQL (CLAS): 18283 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS): 17309 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS): 20979 tokens
+- 16_Data_Types_and_Objects__P02 (DOC): 17891 tokens
--- a/train/data/stats.json
+++ b/train/data/stats.json
@@ -0,0 +1,370 @@
+{
+ "corpus": "/Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl",
+ "seed": 20261003,
+ "max_len": 16384,
+ "estimate_tokens": 2636112,
+ "real_tokens": 2806501,
+ "ratio_real_to_estimate": 1.065,
+ "all": {
+  "docs": 370,
+  "tokens": 2806501,
+  "min": 49,
+  "p50": 5195,
+  "p90": 16923,
+  "p99": 21483,
+  "max": 31552
+ },
+ "kept": {
+  "docs": 325,
+  "tokens": 1966365,
+  "min": 49,
+  "p50": 4027,
+  "p90": 14998,
+  "p99": 16217,
+  "max": 16273
+ },
+ "train": {
+  "docs": 309,
+  "tokens": 1886475,
+  "min": 49,
+  "p50": 4403,
+  "p90": 14998,
+  "p99": 16217,
+  "max": 16273
+ },
+ "valid": {
+  "docs": 16,
+  "tokens": 79890,
+  "min": 130,
+  "p50": 3392,
+  "p90": 16027,
+  "p99": 16129,
+  "max": 16129
+ },
+ "removed": [
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_OBJECTS (CLAS)"
+   ],
+   "tokens": 17042
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_STRUCTURES (CLAS)"
+   ],
+   "tokens": 16628
+  },
+  {
+   "objects": [
+    "32_Performance_Notes__P03 (DOC)"
+   ],
+   "tokens": 16742
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)"
+   ],
+   "tokens": 19117
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)"
+   ],
+   "tokens": 18883
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS)"
+   ],
+   "tokens": 19801
+  },
+  {
+   "objects": [
+    "23_Date_and_Time (DOC)"
+   ],
+   "tokens": 20347
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS)"
+   ],
+   "tokens": 17477
+  },
+  {
+   "objects": [
+    "07_String_Processing__P02 (DOC)"
+   ],
+   "tokens": 17916
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS)"
+   ],
+   "tokens": 17368
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS)"
+   ],
+   "tokens": 31552
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS)"
+   ],
+   "tokens": 19499
+  },
+  {
+   "objects": [
+    "04_ABAP_Object_Orientation__P06 (DOC)"
+   ],
+   "tokens": 16694
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS)"
+   ],
+   "tokens": 18368
+  },
+  {
+   "objects": [
+    "06_Dynamic_Programming__P07 (DOC)"
+   ],
+   "tokens": 23994
+  },
+  {
+   "objects": [
+    "28_Regular_Expressions (DOC)"
+   ],
+   "tokens": 17900
+  },
+  {
+   "objects": [
+    "06_Dynamic_Programming__P06 (DOC)"
+   ],
+   "tokens": 16923
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS)"
+   ],
+   "tokens": 19769
+  },
+  {
+   "objects": [
+    "05_Constructor_Expressions__P01 (DOC)"
+   ],
+   "tokens": 17349
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS)"
+   ],
+   "tokens": 17237
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS)"
+   ],
+   "tokens": 18069
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
+   ],
+   "tokens": 17923
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
+   ],
+   "tokens": 16924
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
+   ],
+   "tokens": 16770
+  },
+  {
+   "objects": [
+    "14_ABAP_Unit_Tests__P03 (DOC)"
+   ],
+   "tokens": 18549
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS)"
+   ],
+   "tokens": 22096
+  },
+  {
+   "objects": [
+    "22_Released_ABAP_Classes__P01 (DOC)"
+   ],
+   "tokens": 19022
+  },
+  {
+   "objects": [
+    "21_XML_JSON__P01 (DOC)"
+   ],
+   "tokens": 16625
+  },
+  {
+   "objects": [
+    "29_Numeric_Operations__P01 (DOC)"
+   ],
+   "tokens": 17805
+  },
+  {
+   "objects": [
+    "03_ABAP_SQL__P01 (DOC)"
+   ],
+   "tokens": 17305
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS)"
+   ],
+   "tokens": 18308
+  },
+  {
+   "objects": [
+    "14_ABAP_Unit_Tests__P04 (DOC)"
+   ],
+   "tokens": 17633
+  },
+  {
+   "objects": [
+    "24_Builtin_Functions (DOC)"
+   ],
+   "tokens": 16851
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)"
+   ],
+   "tokens": 19740
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS)"
+   ],
+   "tokens": 17355
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS)"
+   ],
+   "tokens": 20446
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_STRING_PROC (CLAS)"
+   ],
+   "tokens": 20311
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS)"
+   ],
+   "tokens": 16394
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS)"
+   ],
+   "tokens": 21483
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DATE_TIME (CLAS)"
+   ],
+   "tokens": 18634
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_XML_JSON (CLAS)"
+   ],
+   "tokens": 16825
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_SQL (CLAS)"
+   ],
+   "tokens": 18283
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS)"
+   ],
+   "tokens": 17309
+  },
+  {
+   "objects": [
+    "ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS)"
+   ],
+   "tokens": 20979
+  },
+  {
+   "objects": [
+    "16_Data_Types_and_Objects__P02 (DOC)"
+   ],
+   "tokens": 17891
+  }
+ ],
+ "by_type": {
+  "CLAS": {
+   "docs": 147,
+   "tokens": 827969
+  },
+  "mixed(4)": {
+   "docs": 6,
+   "tokens": 28405
+  },
+  "DOC": {
+   "docs": 106,
+   "tokens": 1012152
+  },
+  "INTF": {
+   "docs": 7,
+   "tokens": 1792
+  },
+  "BDEF": {
+   "docs": 6,
+   "tokens": 1564
+  },
+  "DDLS": {
+   "docs": 27,
+   "tokens": 20614
+  },
+  "mixed(3)": {
+   "docs": 2,
+   "tokens": 6667
+  },
+  "PROG": {
+   "docs": 13,
+   "tokens": 55365
+  },
+  "DDLX": {
+   "docs": 6,
+   "tokens": 5003
+  },
+  "CLAS+DDLS": {
+   "docs": 1,
+   "tokens": 4563
+  },
+  "DCLS": {
+   "docs": 2,
+   "tokens": 245
+  },
+  "FUGR": {
+   "docs": 2,
+   "tokens": 2026
+  }
+ },
+ "source": [
+  "abap-cheat-sheets"
+ ],
+ "iterations_2_epochs": 618
+}
--- a/train/prepare.py
+++ b/train/prepare.py
@@ -0,0 +1,113 @@
+"""Stage 1 data preparation: real token counts, length filter, split by document, report.
+
+  train/.venv/bin/python train/prepare.py [--corpus PATH] [--max-len 16384] [--seed 20261003]
+
+Output: train/data/train.jsonl, valid.jsonl, test.jsonl (copy of valid: `mlx_lm.lora --test` needs a test file),
+train/data/report.md, train/data/stats.json. Only the tokenizer is loaded (no model).
+"""
+import argparse
+import collections
+import json
+import os
+import random
+import re
+
+from transformers import AutoTokenizer
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+OUT = os.path.join(ROOT, "train", "data")
+
+
+def pct(v, p):
+    return v[min(len(v) - 1, int(len(v) * p))]
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--corpus", default=os.path.expanduser("~/projects/abap-llm/corpus/corpus.jsonl"))
+    ap.add_argument("--model", default=os.path.expanduser("~/models/Qwen3.8-27B-4bit"))
+    ap.add_argument("--max-len", type=int, default=16384)
+    ap.add_argument("--seed", type=int, default=20261003)
+    ap.add_argument("--valid-frac", type=float, default=0.05)
+    a = ap.parse_args()
+
+    tok = AutoTokenizer.from_pretrained(a.model)
+    docs = [json.loads(l) for l in open(a.corpus)]
+    for i, d in enumerate(docs):
+        d["_id"] = i
+        d["real_tokens"] = len(tok(d["text"], add_special_tokens=False)["input_ids"])
+    est = sum(d["tokens"] for d in docs)
+    real = sum(d["real_tokens"] for d in docs)
+    kept = [d for d in docs if d["real_tokens"] <= a.max_len]
+    removed = [d for d in docs if d["real_tokens"] > a.max_len]
+
+    # Family = same demo in several release versions (__V755 ...) or parts of one text (__P01 ...). A family
+    # is never split between train and valid (near-duplicates would make the valid loss too good).
+    fam = collections.defaultdict(list)
+    for d in kept:
+        fam[re.sub(r"__(V\d+|P\d+)", "", d["objects"][0] if d["objects"] else str(d["_id"]))].append(d)
+    keys = sorted(fam)
+    random.Random(a.seed).shuffle(keys)
+    n_valid, valid, train = max(1, round(len(kept) * a.valid_frac)), [], []
+    for k in keys:
+        (valid if len(valid) < n_valid else train).extend(fam[k])
+
+    os.makedirs(OUT, exist_ok=True)
+    for name, part in (("train", train), ("valid", valid), ("test", valid)):
+        with open(os.path.join(OUT, name + ".jsonl"), "w") as f:
+            for d in part:
+                f.write(json.dumps({"text": d["text"]}, ensure_ascii=False) + "\n")
+
+    def summ(ds):
+        v = sorted(d["real_tokens"] for d in ds)
+        return {"docs": len(ds), "tokens": sum(v), "min": v[0], "p50": pct(v, .5), "p90": pct(v, .9),
+                "p99": pct(v, .99), "max": v[-1]} if v else {"docs": 0, "tokens": 0}
+
+    by_type = collections.defaultdict(lambda: [0, 0])
+    for d in kept:
+        k = "+".join(d["types"]) if len(d["types"]) <= 2 else "mixed(%d)" % len(d["types"])
+        by_type[k][0] += 1
+        by_type[k][1] += d["real_tokens"]
+    bins = [512, 1024, 2048, 4096, 8192, 16384, 10 ** 9]
+    hist = collections.Counter()
+    for d in docs:
+        for b in bins:
+            if d["real_tokens"] <= b:
+                hist[b] += 1
+                break
+    stats = {"corpus": a.corpus, "seed": a.seed, "max_len": a.max_len, "estimate_tokens": est,
+             "real_tokens": real, "ratio_real_to_estimate": round(real / est, 3), "all": summ(docs),
+             "kept": summ(kept), "train": summ(train), "valid": summ(valid),
+             "removed": [{"objects": d["objects"][:3], "tokens": d["real_tokens"]} for d in removed],
+             "by_type": {k: {"docs": v[0], "tokens": v[1]} for k, v in by_type.items()},
+             "source": sorted({d.get("source", "?") for d in docs}),
+             "iterations_2_epochs": len(train) * 2}
+    json.dump(stats, open(os.path.join(OUT, "stats.json"), "w"), indent=1)
+
+    L = ["# Stage 1 data report", "",
+         f"Corpus: `{a.corpus}` (source {', '.join(stats['source'])}, Apache-2.0, see corpus/out/ATTRIBUTION.md).",
+         f"Tokenizer: `{a.model}`. Seed {a.seed}. max_seq_length {a.max_len}.",
+         f"Split by family (release versions and parts of one document stay together): {len(fam)} families.", "",
+         "| | docs | tokens |", "|---|---|---|",
+         f"| corpus | {len(docs)} | {real} (estimate chars/4: {est}, ratio {stats['ratio_real_to_estimate']}) |",
+         f"| removed (> {a.max_len}) | {len(removed)} | {sum(d['real_tokens'] for d in removed)} |",
+         f"| train | {len(train)} | {stats['train']['tokens']} |",
+         f"| valid (= test file) | {len(valid)} | {stats['valid']['tokens']} |", "",
+         f"Iterations for 2 epochs (batch 1): {len(train) * 2}.", "",
+         "## Token distribution per document (real tokenizer)", "", "| set | min | p50 | p90 | p99 | max |", "|---|---|---|---|---|---|"]
+    for n in ("all", "kept", "train", "valid"):
+        s = stats[n]
+        L.append(f"| {n} | {s['min']} | {s['p50']} | {s['p90']} | {s['p99']} | {s['max']} |")
+    L += ["", "| tokens per document up to | docs |", "|---|---|"]
+    L += [f"| {b if b < 10 ** 9 else 'more'} | {hist[b]} |" for b in bins]
+    L += ["", "## By object type (kept documents)", "", "| types | docs | tokens |", "|---|---|---|"]
+    for k, v in sorted(by_type.items(), key=lambda kv: -kv[1][1]):
+        L.append(f"| {k} | {v[0]} | {v[1]} |")
+    L += ["", "## Removed documents", ""]
+    L += [f"- {', '.join(d['objects'][:2])}: {d['real_tokens']} tokens" for d in removed] or ["None."]
+    open(os.path.join(OUT, "report.md"), "w").write("\n".join(L) + "\n")
+    print("\n".join(L))
+
+
+if __name__ == "__main__":
+    main()