Stage 1 step 1: prepare.py, real corpus numbers (SAP-samples/abap-cheat-sheets)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014aUaQeLnwbb1zTpN7kHeat

train/data/report.md

@@ -0,0 +1,98 @@
+# Stage 1 data report
+
+Corpus: `/Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl` (source abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md).
+Tokenizer: `/Users/erhankeseli/models/Qwen3.8-27B-4bit`. Seed 20261003. max_seq_length 16384.
+Split by family (release versions and parts of one document stay together): 180 families.
+
+| | docs | tokens |
+|---|---|---|
+| corpus | 370 | 2806501 (estimate chars/4: 2636112, ratio 1.065) |
+| removed (> 16384) | 45 | 840136 |
+| train | 309 | 1886475 |
+| valid (= test file) | 16 | 79890 |
+
+Iterations for 2 epochs (batch 1): 618.
+
+## Token distribution per document (real tokenizer)
+
+| set | min | p50 | p90 | p99 | max |
+|---|---|---|---|---|---|
+| all | 49 | 5195 | 16923 | 21483 | 31552 |
+| kept | 49 | 4027 | 14998 | 16217 | 16273 |
+| train | 49 | 4403 | 14998 | 16217 | 16273 |
+| valid | 130 | 3392 | 16027 | 16129 | 16129 |
+
+| tokens per document up to | docs |
+|---|---|
+| 512 | 48 |
+| 1024 | 25 |
+| 2048 | 40 |
+| 4096 | 51 |
+| 8192 | 53 |
+| 16384 | 108 |
+| more | 45 |
+
+## By object type (kept documents)
+
+| types | docs | tokens |
+|---|---|---|
+| DOC | 106 | 1012152 |
+| CLAS | 147 | 827969 |
+| PROG | 13 | 55365 |
+| mixed(4) | 6 | 28405 |
+| DDLS | 27 | 20614 |
+| mixed(3) | 2 | 6667 |
+| DDLX | 6 | 5003 |
+| CLAS+DDLS | 1 | 4563 |
+| FUGR | 2 | 2026 |
+| INTF | 7 | 1792 |
+| BDEF | 6 | 1564 |
+| DCLS | 2 | 245 |
+
+## Removed documents
+
+- ZCL_DEMO_ABAP_OBJECTS (CLAS): 17042 tokens
+- ZCL_DEMO_ABAP_STRUCTURES (CLAS): 16628 tokens
+- 32_Performance_Notes__P03 (DOC): 16742 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 19117 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 18883 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS): 19801 tokens
+- 23_Date_and_Time (DOC): 20347 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS): 17477 tokens
+- 07_String_Processing__P02 (DOC): 17916 tokens
+- ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS): 17368 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS): 31552 tokens
+- ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS): 19499 tokens
+- 04_ABAP_Object_Orientation__P06 (DOC): 16694 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS): 18368 tokens
+- 06_Dynamic_Programming__P07 (DOC): 23994 tokens
+- 28_Regular_Expressions (DOC): 17900 tokens
+- 06_Dynamic_Programming__P06 (DOC): 16923 tokens
+- ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS): 19769 tokens
+- 05_Constructor_Expressions__P01 (DOC): 17349 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS): 17237 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS): 18069 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 17923 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16924 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16770 tokens
+- 14_ABAP_Unit_Tests__P03 (DOC): 18549 tokens
+- ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS): 22096 tokens
+- 22_Released_ABAP_Classes__P01 (DOC): 19022 tokens
+- 21_XML_JSON__P01 (DOC): 16625 tokens
+- 29_Numeric_Operations__P01 (DOC): 17805 tokens
+- 03_ABAP_SQL__P01 (DOC): 17305 tokens
+- ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS): 18308 tokens
+- 14_ABAP_Unit_Tests__P04 (DOC): 17633 tokens
+- 24_Builtin_Functions (DOC): 16851 tokens
+- ZCL_DEMO_ABAP_NUMERIC_OP (CLAS): 19740 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS): 17355 tokens
+- ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS): 20446 tokens
+- ZCL_DEMO_ABAP_STRING_PROC (CLAS): 20311 tokens
+- ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS): 16394 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS): 21483 tokens
+- ZCL_DEMO_ABAP_DATE_TIME (CLAS): 18634 tokens
+- ZCL_DEMO_ABAP_XML_JSON (CLAS): 16825 tokens
+- ZCL_DEMO_ABAP_SQL (CLAS): 18283 tokens
+- ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS): 17309 tokens
+- ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS): 20979 tokens
+- 16_Data_Types_and_Objects__P02 (DOC): 17891 tokens