abap-llm/train/data/report.md

Stage 1 data report

Corpus: /Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl (SAP-samples/abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md). Tokenizer: /Users/erhankeseli/models/Qwen3.8-27B-4bit. Seed 20261003. max_seq_length 16384.

Version dedup (keep the newest; keep an older version only if it differs by more than 5 % of lines)

• Object families: 269; with more than one version: 39.
• Versions kept: 319. Versions removed: 16.
• Records: 370 before, 351 after.
• Tokens before: 2806501 (chars/4 estimate 2636112). After dedup: 2699110. After splitting: 2701996.
• Info: a stricter rule (compare also with the other kept versions) would remove 8 more kept versions.

Splitting of documents over the limit

• Documents split: 43; pieces: 86; hard line splits (a block was too big): 17; pieces still over the limit: 0.

Result

	docs	tokens
train	374	2567180
valid (= test file)	20	134816

Split by family: 184 families. Iterations for 2 epochs (batch 1): 748.

Token share by type

	before dedup	final (kept pieces)
DOC	1301698 (46.4 %)	1302728 (48.2 %)
CLAS	1378559 (49.1 %)	1297878 (48.0 %)
other	126244 (4.5 %)	101390 (3.8 %)

Token distribution per document (final)

set	min	p50	p90	p99	max
all	49	4904	15608	16217	16273
train	49	4904	15638	16177	16252
valid	92	7589	15317	16273	16273

By object type (final)

types	docs	tokens
DOC	138	1302728
CLAS	194	1297878
PROG	11	40071
mixed(4)	6	28405
DDLS	21	11549
mixed(3)	2	6667
DDLX	6	5003
CLAS+DDLS	1	4563
FUGR	2	2026
INTF	7	1792
BDEF	4	1069
DCLS	2	245

Removed versions

object	version	newest	diff	tokens
ZBP_DEMO_ABAP_RAP_RO_U (CLAS)	v758	main	0.015	10696
ZDEMO_ABAP_RAP_RO_M (BDEF)	v756	main	0.037	259
ZDEMO_ABAP_RAP_RO_U (BDEF)	v756	main	0.043	236
ZDEMO_ABAP_CDS_VE_ASSOC_E (DDLS)	v757	main	0.026	542
ZDEMO_ABAP_CDS_VE_JOINS (DDLS)	v757	main	0.009	1827
ZDEMO_ABAP_CDS_VE_AGG_EXP (DDLS)	v757	main	0.019	804
ZDEMO_ABAP_CDS_VE_SEL (DDLS)	v757	main	0.029	3273
ZDEMO_ABAP_CDS_VE_ASSOC (DDLS)	v757	main	0.008	2451
ZDEMO_ABAP_FLI_VE (DDLS)	v816	main	0.0	168
ZCL_DEMO_ABAP_DTYPE_DOBJ (CLAS)	v816	main	0.009	27407
ZCL_DEMO_ABAP_DISPLAY (CLAS)	v755	v757	0.02	1677
ZCL_DEMO_ABAP_UNIT_TEST (CLAS)	v758	main	0.034	19799
ZDEMO_ABAP_DYNPRO (PROG)	v757	v816	0.001	8026
ZBP_DEMO_ABAP_RAP_DRAFT_M (CLAS)	v757	v758	0.005	2474
ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)	v816	main	0.003	20484
ZDEMO_ABAP_ALV (PROG)	v757	v816	0.002	7268

Split documents

• ['ZCL_DEMO_ABAP_OBJECTS (CLAS)']: 17042 tokens -> 2 pieces [14777, 2342]
• ['ZCL_DEMO_ABAP_STRUCTURES (CLAS)']: 16628 tokens -> 2 pieces [14200, 2503]
• ['32_Performance_Notes__P03 (DOC)']: 16742 tokens -> 2 pieces [15846, 961]
• ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 19117 tokens -> 2 pieces [14425, 4770]
• ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 18883 tokens -> 2 pieces [14607, 4324]
• ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS)']: 19801 tokens -> 2 pieces [15826, 4036]
• ['23_Date_and_Time (DOC)']: 20347 tokens -> 2 pieces [10310, 10098]
• ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS)']: 17477 tokens -> 2 pieces [2125, 15451]
• ['07_String_Processing__P02 (DOC)']: 17916 tokens -> 2 pieces [11096, 6879]
• ['ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS)']: 17368 tokens -> 2 pieces [14031, 3415]
• ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS)']: 31552 tokens -> 2 pieces [15833, 15780]
• ['ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS)']: 19499 tokens -> 2 pieces [15831, 3729]
• ['04_ABAP_Object_Orientation__P06 (DOC)']: 16694 tokens -> 2 pieces [9045, 7716]
• ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS)']: 18368 tokens -> 2 pieces [15833, 2600]
• ['06_Dynamic_Programming__P07 (DOC)']: 23994 tokens -> 2 pieces [15836, 8221]
• ['28_Regular_Expressions (DOC)']: 17900 tokens -> 2 pieces [15031, 2930]
• ['06_Dynamic_Programming__P06 (DOC)']: 16923 tokens -> 2 pieces [15842, 1144]
• ['05_Constructor_Expressions__P01 (DOC)']: 17349 tokens -> 2 pieces [15147, 2265]
• ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS)']: 17237 tokens -> 2 pieces [15833, 1469]
• ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS)']: 18069 tokens -> 2 pieces [15842, 2292]
• ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 17923 tokens -> 2 pieces [15329, 2672]
• ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16924 tokens -> 2 pieces [15749, 1223]
• ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16770 tokens -> 2 pieces [14210, 2608]
• ['14_ABAP_Unit_Tests__P03 (DOC)']: 18549 tokens -> 2 pieces [15834, 2782]
• ['ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS)']: 22096 tokens -> 2 pieces [15814, 6342]
• ['22_Released_ABAP_Classes__P01 (DOC)']: 19022 tokens -> 2 pieces [14661, 4430]
• ['21_XML_JSON__P01 (DOC)']: 16625 tokens -> 2 pieces [9855, 6839]
• ['29_Numeric_Operations__P01 (DOC)']: 17805 tokens -> 2 pieces [15849, 2025]
• ['03_ABAP_SQL__P01 (DOC)']: 17305 tokens -> 2 pieces [14168, 3198]
• ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS)']: 18308 tokens -> 2 pieces [15842, 2531]
• ['14_ABAP_Unit_Tests__P04 (DOC)']: 17633 tokens -> 2 pieces [15834, 1866]
• ['24_Builtin_Functions (DOC)']: 16851 tokens -> 2 pieces [12541, 4365]
• ['ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)']: 19740 tokens -> 2 pieces [15825, 3964]
• ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS)']: 17355 tokens -> 2 pieces [15832, 1584]
• ['ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS)']: 20446 tokens -> 2 pieces [15454, 5071]
• ['ZCL_DEMO_ABAP_STRING_PROC (CLAS)']: 20311 tokens -> 2 pieces [15590, 4797]
• ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS)']: 21483 tokens -> 2 pieces [15815, 5730]
• ['ZCL_DEMO_ABAP_DATE_TIME (CLAS)']: 18634 tokens -> 2 pieces [15577, 3138]
• ['ZCL_DEMO_ABAP_XML_JSON (CLAS)']: 16825 tokens -> 2 pieces [15753, 1154]
• ['ZCL_DEMO_ABAP_SQL (CLAS)']: 18283 tokens -> 2 pieces [12526, 5831]
• ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS)']: 17309 tokens -> 2 pieces [2124, 15283]
• ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS)']: 20979 tokens -> 2 pieces [15831, 5210]
• ['16_Data_Types_and_Objects__P02 (DOC)']: 17891 tokens -> 2 pieces [14421, 3541]