# Stage 1 data report

Corpus: `/Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl` (SAP-samples/abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md).
Tokenizer: `/Users/erhankeseli/models/Qwen3.8-27B-4bit`. Seed 20261003. max_seq_length 16384.

## Version dedup (keep the newest; keep an older version only if it differs by more than 5 % of lines)

- Object families: 269; with more than one version: 39.
- Versions kept: 319. Versions removed: 16.
- Records: 370 before, 351 after.
- Tokens before: 2806501 (chars/4 estimate 2636112). After dedup: 2699110. After splitting: 2701996.
- Info: a stricter rule (compare also with the other kept versions) would remove 8 more kept versions.

## Splitting of documents over the limit

- Documents split: 43; pieces: 86; hard line splits (a block was too big): 17; pieces still over the limit: 0.

## Result

| | docs | tokens |
|---|---|---|
| train | 374 | 2567180 |
| valid (= test file) | 20 | 134816 |

Split by family: 184 families. **Iterations for 2 epochs (batch 1): 748.**

## Token share by type

| | before dedup | final (kept pieces) |
|---|---|---|
| DOC | 1301698 (46.4 %) | 1302728 (48.2 %) |
| CLAS | 1378559 (49.1 %) | 1297878 (48.0 %) |
| other | 126244 (4.5 %) | 101390 (3.8 %) |

## Token distribution per document (final)

| set | min | p50 | p90 | p99 | max |
|---|---|---|---|---|---|
| all | 49 | 4904 | 15608 | 16217 | 16273 |
| train | 49 | 4904 | 15638 | 16177 | 16252 |
| valid | 92 | 7589 | 15317 | 16273 | 16273 |

## By object type (final)

| types | docs | tokens |
|---|---|---|
| DOC | 138 | 1302728 |
| CLAS | 194 | 1297878 |
| PROG | 11 | 40071 |
| mixed(4) | 6 | 28405 |
| DDLS | 21 | 11549 |
| mixed(3) | 2 | 6667 |
| DDLX | 6 | 5003 |
| CLAS+DDLS | 1 | 4563 |
| FUGR | 2 | 2026 |
| INTF | 7 | 1792 |
| BDEF | 4 | 1069 |
| DCLS | 2 | 245 |

## Removed versions

| object | version | newest | diff | tokens |
|---|---|---|---|---|
| ZBP_DEMO_ABAP_RAP_RO_U (CLAS) | v758 | main | 0.015 | 10696 |
| ZDEMO_ABAP_RAP_RO_M (BDEF) | v756 | main | 0.037 | 259 |
| ZDEMO_ABAP_RAP_RO_U (BDEF) | v756 | main | 0.043 | 236 |
| ZDEMO_ABAP_CDS_VE_ASSOC_E (DDLS) | v757 | main | 0.026 | 542 |
| ZDEMO_ABAP_CDS_VE_JOINS (DDLS) | v757 | main | 0.009 | 1827 |
| ZDEMO_ABAP_CDS_VE_AGG_EXP (DDLS) | v757 | main | 0.019 | 804 |
| ZDEMO_ABAP_CDS_VE_SEL (DDLS) | v757 | main | 0.029 | 3273 |
| ZDEMO_ABAP_CDS_VE_ASSOC (DDLS) | v757 | main | 0.008 | 2451 |
| ZDEMO_ABAP_FLI_VE (DDLS) | v816 | main | 0.0 | 168 |
| ZCL_DEMO_ABAP_DTYPE_DOBJ (CLAS) | v816 | main | 0.009 | 27407 |
| ZCL_DEMO_ABAP_DISPLAY (CLAS) | v755 | v757 | 0.02 | 1677 |
| ZCL_DEMO_ABAP_UNIT_TEST (CLAS) | v758 | main | 0.034 | 19799 |
| ZDEMO_ABAP_DYNPRO (PROG) | v757 | v816 | 0.001 | 8026 |
| ZBP_DEMO_ABAP_RAP_DRAFT_M (CLAS) | v757 | v758 | 0.005 | 2474 |
| ZCL_DEMO_ABAP_NUMERIC_OP (CLAS) | v816 | main | 0.003 | 20484 |
| ZDEMO_ABAP_ALV (PROG) | v757 | v816 | 0.002 | 7268 |

## Split documents

- ['ZCL_DEMO_ABAP_OBJECTS (CLAS)']: 17042 tokens -> 2 pieces [14777, 2342]
- ['ZCL_DEMO_ABAP_STRUCTURES (CLAS)']: 16628 tokens -> 2 pieces [14200, 2503]
- ['32_Performance_Notes__P03 (DOC)']: 16742 tokens -> 2 pieces [15846, 961]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 19117 tokens -> 2 pieces [14425, 4770]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 18883 tokens -> 2 pieces [14607, 4324]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS)']: 19801 tokens -> 2 pieces [15826, 4036]
- ['23_Date_and_Time (DOC)']: 20347 tokens -> 2 pieces [10310, 10098]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS)']: 17477 tokens -> 2 pieces [2125, 15451]
- ['07_String_Processing__P02 (DOC)']: 17916 tokens -> 2 pieces [11096, 6879]
- ['ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS)']: 17368 tokens -> 2 pieces [14031, 3415]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS)']: 31552 tokens -> 2 pieces [15833, 15780]
- ['ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS)']: 19499 tokens -> 2 pieces [15831, 3729]
- ['04_ABAP_Object_Orientation__P06 (DOC)']: 16694 tokens -> 2 pieces [9045, 7716]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS)']: 18368 tokens -> 2 pieces [15833, 2600]
- ['06_Dynamic_Programming__P07 (DOC)']: 23994 tokens -> 2 pieces [15836, 8221]
- ['28_Regular_Expressions (DOC)']: 17900 tokens -> 2 pieces [15031, 2930]
- ['06_Dynamic_Programming__P06 (DOC)']: 16923 tokens -> 2 pieces [15842, 1144]
- ['05_Constructor_Expressions__P01 (DOC)']: 17349 tokens -> 2 pieces [15147, 2265]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS)']: 17237 tokens -> 2 pieces [15833, 1469]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS)']: 18069 tokens -> 2 pieces [15842, 2292]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 17923 tokens -> 2 pieces [15329, 2672]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16924 tokens -> 2 pieces [15749, 1223]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16770 tokens -> 2 pieces [14210, 2608]
- ['14_ABAP_Unit_Tests__P03 (DOC)']: 18549 tokens -> 2 pieces [15834, 2782]
- ['ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS)']: 22096 tokens -> 2 pieces [15814, 6342]
- ['22_Released_ABAP_Classes__P01 (DOC)']: 19022 tokens -> 2 pieces [14661, 4430]
- ['21_XML_JSON__P01 (DOC)']: 16625 tokens -> 2 pieces [9855, 6839]
- ['29_Numeric_Operations__P01 (DOC)']: 17805 tokens -> 2 pieces [15849, 2025]
- ['03_ABAP_SQL__P01 (DOC)']: 17305 tokens -> 2 pieces [14168, 3198]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS)']: 18308 tokens -> 2 pieces [15842, 2531]
- ['14_ABAP_Unit_Tests__P04 (DOC)']: 17633 tokens -> 2 pieces [15834, 1866]
- ['24_Builtin_Functions (DOC)']: 16851 tokens -> 2 pieces [12541, 4365]
- ['ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)']: 19740 tokens -> 2 pieces [15825, 3964]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS)']: 17355 tokens -> 2 pieces [15832, 1584]
- ['ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS)']: 20446 tokens -> 2 pieces [15454, 5071]
- ['ZCL_DEMO_ABAP_STRING_PROC (CLAS)']: 20311 tokens -> 2 pieces [15590, 4797]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS)']: 21483 tokens -> 2 pieces [15815, 5730]
- ['ZCL_DEMO_ABAP_DATE_TIME (CLAS)']: 18634 tokens -> 2 pieces [15577, 3138]
- ['ZCL_DEMO_ABAP_XML_JSON (CLAS)']: 16825 tokens -> 2 pieces [15753, 1154]
- ['ZCL_DEMO_ABAP_SQL (CLAS)']: 18283 tokens -> 2 pieces [12526, 5831]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS)']: 17309 tokens -> 2 pieces [2124, 15283]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS)']: 20979 tokens -> 2 pieces [15831, 5210]
- ['16_Data_Types_and_Objects__P02 (DOC)']: 17891 tokens -> 2 pieces [14421, 3541]