Stage 1 data report
Corpus: /Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl (source abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md).
Tokenizer: /Users/erhankeseli/models/Qwen3.8-27B-4bit. Seed 20261003. max_seq_length 16384.
Split by family (release versions and parts of one document stay together): 180 families.
|
docs |
tokens |
| corpus |
370 |
2806501 (estimate chars/4: 2636112, ratio 1.065) |
| removed (> 16384) |
45 |
840136 |
| train |
309 |
1886475 |
| valid (= test file) |
16 |
79890 |
Iterations for 2 epochs (batch 1): 618.
Token distribution per document (real tokenizer)
| set |
min |
p50 |
p90 |
p99 |
max |
| all |
49 |
5195 |
16923 |
21483 |
31552 |
| kept |
49 |
4027 |
14998 |
16217 |
16273 |
| train |
49 |
4403 |
14998 |
16217 |
16273 |
| valid |
130 |
3392 |
16027 |
16129 |
16129 |
| tokens per document up to |
docs |
| 512 |
48 |
| 1024 |
25 |
| 2048 |
40 |
| 4096 |
51 |
| 8192 |
53 |
| 16384 |
108 |
| more |
45 |
By object type (kept documents)
| types |
docs |
tokens |
| DOC |
106 |
1012152 |
| CLAS |
147 |
827969 |
| PROG |
13 |
55365 |
| mixed(4) |
6 |
28405 |
| DDLS |
27 |
20614 |
| mixed(3) |
2 |
6667 |
| DDLX |
6 |
5003 |
| CLAS+DDLS |
1 |
4563 |
| FUGR |
2 |
2026 |
| INTF |
7 |
1792 |
| BDEF |
6 |
1564 |
| DCLS |
2 |
245 |
Removed documents
- ZCL_DEMO_ABAP_OBJECTS (CLAS): 17042 tokens
- ZCL_DEMO_ABAP_STRUCTURES (CLAS): 16628 tokens
- 32_Performance_Notes__P03 (DOC): 16742 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 19117 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 18883 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS): 19801 tokens
- 23_Date_and_Time (DOC): 20347 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS): 17477 tokens
- 07_String_Processing__P02 (DOC): 17916 tokens
- ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS): 17368 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS): 31552 tokens
- ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS): 19499 tokens
- 04_ABAP_Object_Orientation__P06 (DOC): 16694 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS): 18368 tokens
- 06_Dynamic_Programming__P07 (DOC): 23994 tokens
- 28_Regular_Expressions (DOC): 17900 tokens
- 06_Dynamic_Programming__P06 (DOC): 16923 tokens
- ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS): 19769 tokens
- 05_Constructor_Expressions__P01 (DOC): 17349 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS): 17237 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS): 18069 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 17923 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16924 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16770 tokens
- 14_ABAP_Unit_Tests__P03 (DOC): 18549 tokens
- ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS): 22096 tokens
- 22_Released_ABAP_Classes__P01 (DOC): 19022 tokens
- 21_XML_JSON__P01 (DOC): 16625 tokens
- 29_Numeric_Operations__P01 (DOC): 17805 tokens
- 03_ABAP_SQL__P01 (DOC): 17305 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS): 18308 tokens
- 14_ABAP_Unit_Tests__P04 (DOC): 17633 tokens
- 24_Builtin_Functions (DOC): 16851 tokens
- ZCL_DEMO_ABAP_NUMERIC_OP (CLAS): 19740 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS): 17355 tokens
- ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS): 20446 tokens
- ZCL_DEMO_ABAP_STRING_PROC (CLAS): 20311 tokens
- ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS): 16394 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS): 21483 tokens
- ZCL_DEMO_ABAP_DATE_TIME (CLAS): 18634 tokens
- ZCL_DEMO_ABAP_XML_JSON (CLAS): 16825 tokens
- ZCL_DEMO_ABAP_SQL (CLAS): 18283 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS): 17309 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS): 20979 tokens
- 16_Data_Types_and_Objects__P02 (DOC): 17891 tokens
e083c9ca13