diff --git a/docs/stage1-training-task.md b/docs/stage1-training-task.md index b29b009..ec2e9ca 100644 --- a/docs/stage1-training-task.md +++ b/docs/stage1-training-task.md @@ -14,7 +14,7 @@ Stage 2 (SFT on teacher trajectories) is not part of this task. Input: `~/projects/abap-llm/corpus/corpus.jsonl`. Source (since 2026-10-03): SAP-samples/abap-cheat-sheets, Apache-2.0 (`corpus/out/ATTRIBUTION.md`). Real numbers from Step 1 (`train/data/report.md`): 370 records, 2,806,501 tokens (Qwen tokenizer; -the chars/4 estimate is 2.64M). After version dedup and splitting: train 374 documents / 2.57M tokens, +the chars/4 estimate is 2.64M). After version dedup and splitting: train 374 documents / 2.53M tokens, valid 20 documents / 135k tokens, 748 iterations for 2 epochs. The old numbers (1,765 documents, 1.4M tokens) are outdated. One record per line: `{"text", "objects", "types", "language_version", "package_path", "grouped", "obsolete", "tokens"}`. The `tokens` field is an estimate (chars / 4); there is also a field `source`. diff --git a/train/STATE.md b/train/STATE.md index c05dcf4..9acf070 100644 --- a/train/STATE.md +++ b/train/STATE.md @@ -1,6 +1,6 @@ # Stage 1 state -Task: `docs/stage1-training-task.md`. Settings and weights: `train/README.md`. Updated 2026-10-03 22:30. +Task: `docs/stage1-training-task.md`. Settings and weights: `train/README.md`. Updated 2026-10-03 23:00. ## Done @@ -15,14 +15,16 @@ Task: `docs/stage1-training-task.md`. Settings and weights: `train/README.md`. U - Runner: `train/baseline.py --label