erhan/abap-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1ddb9a7c65eff237e2764e5526783d8f3e6027bd
abap-llm/train/data/report.md

5.9 KiB
Raw Blame History

Stage 1 data report

Corpus: /Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl (SAP-samples/abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md). Tokenizer: /Users/erhankeseli/models/Qwen3.8-27B-4bit. Seed 20261003. max_seq_length 16384.

Version dedup (strict rule; an older version is kept only if it differs by more than 5 % of lines)

  • Object families: 269; with more than one version: 39.
  • Versions kept: 323. Versions removed: 12.
  • Records: 370 before, 353 after.
  • Tokens before: 2806501 (chars/4 estimate 2636112). After dedup: 2658082. After splitting: 2660834.
  • Rule: main (Cloud) and the newest v* (Standard ABAP) always kept; each older v* compared with the next newer v*; other branches (0 comparisons) compared with main.

Splitting of documents over the limit

  • Documents split: 41; pieces: 82; hard line splits (a block was too big): 15; pieces still over the limit: 0.

Result

docs tokens
train 374 2525568
valid (= test file) 20 135266

Split by family: 184 families. Iterations for 2 epochs (batch 1): 748.

Token share by type

before dedup final (kept pieces)
DOC 1301698 (46.4 %) 1302728 (49.0 %)
CLAS 1378559 (49.1 %) 1247458 (46.9 %)
other 126244 (4.5 %) 110648 (4.2 %)

Token distribution per document (final)

set min p50 p90 p99 max
all 49 4797 15577 16217 16273
train 49 4770 15590 16177 16252
valid 135 7589 15317 16273 16273

By object type (final)

types docs tokens
DOC 138 1302728
CLAS 187 1247458
PROG 11 40071
mixed(4) 6 28405
DDLS 27 20614
mixed(3) 2 6667
DDLX 6 5003
CLAS+DDLS 1 4563
FUGR 2 2026
INTF 7 1792
BDEF 5 1262
DCLS 2 245

Removed versions

object version newest diff tokens
ZCL_DEMO_ABAP_OBJECTS (CLAS) v755 v757 0.01 14944
ZDEMO_ABAP_RAP_DRAFT_M (BDEF) v756 v757 0.033 302
ZCL_DEMO_ABAP_DTYPE_DOBJ (CLAS) v756 v757 0.015 21263
ZCL_DEMO_ABAP_DTYPE_DOBJ (CLAS) v755 v756 0.047 20431
ZCL_DEMO_ABAP_DISPLAY (CLAS) v755 v757 0.02 1677
ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS) v755 v757 0.01 17309
ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS) v756 v757 0.021 25883
ZCL_DEMO_ABAP_SQL (CLAS) v755 v757 0.035 14949
ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS) v755 v757 0.048 13893
ZDEMO_ABAP_DYNPRO (PROG) v757 v816 0.001 8026
ZBP_DEMO_ABAP_RAP_DRAFT_M (CLAS) v757 v758 0.005 2474
ZDEMO_ABAP_ALV (PROG) v757 v816 0.002 7268

Split documents

  • ['ZCL_DEMO_ABAP_OBJECTS (CLAS)']: 17042 tokens -> 2 pieces [14777, 2342]
  • ['ZCL_DEMO_ABAP_STRUCTURES (CLAS)']: 16628 tokens -> 2 pieces [14200, 2503]
  • ['32_Performance_Notes__P03 (DOC)']: 16742 tokens -> 2 pieces [15846, 961]
  • ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 19117 tokens -> 2 pieces [14425, 4770]
  • ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 18883 tokens -> 2 pieces [14607, 4324]
  • ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS)']: 19801 tokens -> 2 pieces [15826, 4036]
  • ['23_Date_and_Time (DOC)']: 20347 tokens -> 2 pieces [10310, 10098]
  • ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS)']: 17477 tokens -> 2 pieces [2125, 15451]
  • ['07_String_Processing__P02 (DOC)']: 17916 tokens -> 2 pieces [11096, 6879]
  • ['ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS)']: 17368 tokens -> 2 pieces [14031, 3415]
  • ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS)']: 31552 tokens -> 2 pieces [15833, 15780]
  • ['ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS)']: 19499 tokens -> 2 pieces [15831, 3729]
  • ['04_ABAP_Object_Orientation__P06 (DOC)']: 16694 tokens -> 2 pieces [9045, 7716]
  • ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS)']: 18368 tokens -> 2 pieces [15833, 2600]
  • ['06_Dynamic_Programming__P07 (DOC)']: 23994 tokens -> 2 pieces [15836, 8221]
  • ['28_Regular_Expressions (DOC)']: 17900 tokens -> 2 pieces [15031, 2930]
  • ['06_Dynamic_Programming__P06 (DOC)']: 16923 tokens -> 2 pieces [15842, 1144]
  • ['ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS)']: 19769 tokens -> 2 pieces [15819, 4012]
  • ['05_Constructor_Expressions__P01 (DOC)']: 17349 tokens -> 2 pieces [15147, 2265]
  • ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 17923 tokens -> 2 pieces [15329, 2672]
  • ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16924 tokens -> 2 pieces [15749, 1223]
  • ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16770 tokens -> 2 pieces [14210, 2608]
  • ['14_ABAP_Unit_Tests__P03 (DOC)']: 18549 tokens -> 2 pieces [15834, 2782]
  • ['ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS)']: 22096 tokens -> 2 pieces [15814, 6342]
  • ['22_Released_ABAP_Classes__P01 (DOC)']: 19022 tokens -> 2 pieces [14661, 4430]
  • ['21_XML_JSON__P01 (DOC)']: 16625 tokens -> 2 pieces [9855, 6839]
  • ['29_Numeric_Operations__P01 (DOC)']: 17805 tokens -> 2 pieces [15849, 2025]
  • ['03_ABAP_SQL__P01 (DOC)']: 17305 tokens -> 2 pieces [14168, 3198]
  • ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS)']: 18308 tokens -> 2 pieces [15842, 2531]
  • ['14_ABAP_Unit_Tests__P04 (DOC)']: 17633 tokens -> 2 pieces [15834, 1866]
  • ['24_Builtin_Functions (DOC)']: 16851 tokens -> 2 pieces [12541, 4365]
  • ['ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)']: 19740 tokens -> 2 pieces [15825, 3964]
  • ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS)']: 17355 tokens -> 2 pieces [15832, 1584]
  • ['ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS)']: 20446 tokens -> 2 pieces [15454, 5071]
  • ['ZCL_DEMO_ABAP_STRING_PROC (CLAS)']: 20311 tokens -> 2 pieces [15590, 4797]
  • ['ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS)']: 16394 tokens -> 2 pieces [15301, 1187]
  • ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS)']: 21483 tokens -> 2 pieces [15815, 5730]
  • ['ZCL_DEMO_ABAP_DATE_TIME (CLAS)']: 18634 tokens -> 2 pieces [15577, 3138]
  • ['ZCL_DEMO_ABAP_XML_JSON (CLAS)']: 16825 tokens -> 2 pieces [15753, 1154]
  • ['ZCL_DEMO_ABAP_SQL (CLAS)']: 18283 tokens -> 2 pieces [12526, 5831]
  • ['16_Data_Types_and_Objects__P02 (DOC)']: 17891 tokens -> 2 pieces [14421, 3541]
Reference in New Issue View Git Blame Copy Permalink