erhan/abap-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Stage 1 data: version dedup, splitting of long documents, family split; 748 iterations

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014aUaQeLnwbb1zTpN7kHeat
This commit is contained in:
Kral
2026-10-03 22:09:15 +02:00
parent e083c9ca13
commit 0677d035da
5 changed files with 1014 additions and 421 deletions
Download Patch File Download Diff File Expand all files Collapse all files

175
train/data/report.md
View File

@@ -1,98 +1,125 @@
# Stage 1 data report
Corpus: `/Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl` (source abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md).
Corpus: `/Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl` (SAP-samples/abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md).
Tokenizer: `/Users/erhankeseli/models/Qwen3.8-27B-4bit`. Seed 20261003. max_seq_length 16384.
Split by family (release versions and parts of one document stay together): 180 families.
## Version dedup (keep the newest; keep an older version only if it differs by more than 5 % of lines)
- Object families: 269; with more than one version: 39.
- Versions kept: 319. Versions removed: 16.
- Records: 370 before, 351 after.
- Tokens before: 2806501 (chars/4 estimate 2636112). After dedup: 2699110. After splitting: 2701996.
- Info: a stricter rule (compare also with the other kept versions) would remove 8 more kept versions.
## Splitting of documents over the limit
- Documents split: 43; pieces: 86; hard line splits (a block was too big): 17; pieces still over the limit: 0.
## Result
| | docs | tokens |
|---|---|---|
| corpus | 370 | 2806501 (estimate chars/4: 2636112, ratio 1.065) |
| removed (> 16384) | 45 | 840136 |
| train | 309 | 1886475 |
| valid (= test file) | 16 | 79890 |
| train | 374 | 2567180 |
| valid (= test file) | 20 | 134816 |
Iterations for 2 epochs (batch 1): 618.
Split by family: 184 families. **Iterations for 2 epochs (batch 1): 748.**
## Token distribution per document (real tokenizer)
## Token share by type
| | before dedup | final (kept pieces) |
|---|---|---|
| DOC | 1301698 (46.4 %) | 1302728 (48.2 %) |
| CLAS | 1378559 (49.1 %) | 1297878 (48.0 %) |
| other | 126244 (4.5 %) | 101390 (3.8 %) |
## Token distribution per document (final)
| set | min | p50 | p90 | p99 | max |
|---|---|---|---|---|---|
| all | 49 | 5195 | 16923 | 21483 | 31552 |
| kept | 49 | 4027 | 14998 | 16217 | 16273 |
| train | 49 | 4403 | 14998 | 16217 | 16273 |
| valid | 130 | 3392 | 16027 | 16129 | 16129 |
| all | 49 | 4904 | 15608 | 16217 | 16273 |
| train | 49 | 4904 | 15638 | 16177 | 16252 |
| valid | 92 | 7589 | 15317 | 16273 | 16273 |
| tokens per document up to | docs |
|---|---|
| 512 | 48 |
| 1024 | 25 |
| 2048 | 40 |
| 4096 | 51 |
| 8192 | 53 |
| 16384 | 108 |
| more | 45 |
## By object type (kept documents)
## By object type (final)
| types | docs | tokens |
|---|---|---|
| DOC | 106 | 1012152 |
| CLAS | 147 | 827969 |
| PROG | 13 | 55365 |
| DOC | 138 | 1302728 |
| CLAS | 194 | 1297878 |
| PROG | 11 | 40071 |
| mixed(4) | 6 | 28405 |
| DDLS | 27 | 20614 |
| DDLS | 21 | 11549 |
| mixed(3) | 2 | 6667 |
| DDLX | 6 | 5003 |
| CLAS+DDLS | 1 | 4563 |
| FUGR | 2 | 2026 |
| INTF | 7 | 1792 |
| BDEF | 6 | 1564 |
| BDEF | 4 | 1069 |
| DCLS | 2 | 245 |
## Removed documents
## Removed versions
- ZCL_DEMO_ABAP_OBJECTS (CLAS): 17042 tokens
- ZCL_DEMO_ABAP_STRUCTURES (CLAS): 16628 tokens
- 32_Performance_Notes__P03 (DOC): 16742 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 19117 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 18883 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS): 19801 tokens
- 23_Date_and_Time (DOC): 20347 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS): 17477 tokens
- 07_String_Processing__P02 (DOC): 17916 tokens
- ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS): 17368 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS): 31552 tokens
- ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS): 19499 tokens
- 04_ABAP_Object_Orientation__P06 (DOC): 16694 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS): 18368 tokens
- 06_Dynamic_Programming__P07 (DOC): 23994 tokens
- 28_Regular_Expressions (DOC): 17900 tokens
- 06_Dynamic_Programming__P06 (DOC): 16923 tokens
- ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS): 19769 tokens
- 05_Constructor_Expressions__P01 (DOC): 17349 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS): 17237 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS): 18069 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 17923 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16924 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16770 tokens
- 14_ABAP_Unit_Tests__P03 (DOC): 18549 tokens
- ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS): 22096 tokens
- 22_Released_ABAP_Classes__P01 (DOC): 19022 tokens
- 21_XML_JSON__P01 (DOC): 16625 tokens
- 29_Numeric_Operations__P01 (DOC): 17805 tokens
- 03_ABAP_SQL__P01 (DOC): 17305 tokens
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS): 18308 tokens
- 14_ABAP_Unit_Tests__P04 (DOC): 17633 tokens
- 24_Builtin_Functions (DOC): 16851 tokens
- ZCL_DEMO_ABAP_NUMERIC_OP (CLAS): 19740 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS): 17355 tokens
- ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS): 20446 tokens
- ZCL_DEMO_ABAP_STRING_PROC (CLAS): 20311 tokens
- ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS): 16394 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS): 21483 tokens
- ZCL_DEMO_ABAP_DATE_TIME (CLAS): 18634 tokens
- ZCL_DEMO_ABAP_XML_JSON (CLAS): 16825 tokens
- ZCL_DEMO_ABAP_SQL (CLAS): 18283 tokens
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS): 17309 tokens
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS): 20979 tokens
- 16_Data_Types_and_Objects__P02 (DOC): 17891 tokens
| object | version | newest | diff | tokens |
|---|---|---|---|---|
| ZBP_DEMO_ABAP_RAP_RO_U (CLAS) | v758 | main | 0.015 | 10696 |
| ZDEMO_ABAP_RAP_RO_M (BDEF) | v756 | main | 0.037 | 259 |
| ZDEMO_ABAP_RAP_RO_U (BDEF) | v756 | main | 0.043 | 236 |
| ZDEMO_ABAP_CDS_VE_ASSOC_E (DDLS) | v757 | main | 0.026 | 542 |
| ZDEMO_ABAP_CDS_VE_JOINS (DDLS) | v757 | main | 0.009 | 1827 |
| ZDEMO_ABAP_CDS_VE_AGG_EXP (DDLS) | v757 | main | 0.019 | 804 |
| ZDEMO_ABAP_CDS_VE_SEL (DDLS) | v757 | main | 0.029 | 3273 |
| ZDEMO_ABAP_CDS_VE_ASSOC (DDLS) | v757 | main | 0.008 | 2451 |
| ZDEMO_ABAP_FLI_VE (DDLS) | v816 | main | 0.0 | 168 |
| ZCL_DEMO_ABAP_DTYPE_DOBJ (CLAS) | v816 | main | 0.009 | 27407 |
| ZCL_DEMO_ABAP_DISPLAY (CLAS) | v755 | v757 | 0.02 | 1677 |
| ZCL_DEMO_ABAP_UNIT_TEST (CLAS) | v758 | main | 0.034 | 19799 |
| ZDEMO_ABAP_DYNPRO (PROG) | v757 | v816 | 0.001 | 8026 |
| ZBP_DEMO_ABAP_RAP_DRAFT_M (CLAS) | v757 | v758 | 0.005 | 2474 |
| ZCL_DEMO_ABAP_NUMERIC_OP (CLAS) | v816 | main | 0.003 | 20484 |
| ZDEMO_ABAP_ALV (PROG) | v757 | v816 | 0.002 | 7268 |
## Split documents
- ['ZCL_DEMO_ABAP_OBJECTS (CLAS)']: 17042 tokens -> 2 pieces [14777, 2342]
- ['ZCL_DEMO_ABAP_STRUCTURES (CLAS)']: 16628 tokens -> 2 pieces [14200, 2503]
- ['32_Performance_Notes__P03 (DOC)']: 16742 tokens -> 2 pieces [15846, 961]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 19117 tokens -> 2 pieces [14425, 4770]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)']: 18883 tokens -> 2 pieces [14607, 4324]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS)']: 19801 tokens -> 2 pieces [15826, 4036]
- ['23_Date_and_Time (DOC)']: 20347 tokens -> 2 pieces [10310, 10098]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS)']: 17477 tokens -> 2 pieces [2125, 15451]
- ['07_String_Processing__P02 (DOC)']: 17916 tokens -> 2 pieces [11096, 6879]
- ['ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS)']: 17368 tokens -> 2 pieces [14031, 3415]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS)']: 31552 tokens -> 2 pieces [15833, 15780]
- ['ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS)']: 19499 tokens -> 2 pieces [15831, 3729]
- ['04_ABAP_Object_Orientation__P06 (DOC)']: 16694 tokens -> 2 pieces [9045, 7716]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS)']: 18368 tokens -> 2 pieces [15833, 2600]
- ['06_Dynamic_Programming__P07 (DOC)']: 23994 tokens -> 2 pieces [15836, 8221]
- ['28_Regular_Expressions (DOC)']: 17900 tokens -> 2 pieces [15031, 2930]
- ['06_Dynamic_Programming__P06 (DOC)']: 16923 tokens -> 2 pieces [15842, 1144]
- ['05_Constructor_Expressions__P01 (DOC)']: 17349 tokens -> 2 pieces [15147, 2265]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS)']: 17237 tokens -> 2 pieces [15833, 1469]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS)']: 18069 tokens -> 2 pieces [15842, 2292]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 17923 tokens -> 2 pieces [15329, 2672]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16924 tokens -> 2 pieces [15749, 1223]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)']: 16770 tokens -> 2 pieces [14210, 2608]
- ['14_ABAP_Unit_Tests__P03 (DOC)']: 18549 tokens -> 2 pieces [15834, 2782]
- ['ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS)']: 22096 tokens -> 2 pieces [15814, 6342]
- ['22_Released_ABAP_Classes__P01 (DOC)']: 19022 tokens -> 2 pieces [14661, 4430]
- ['21_XML_JSON__P01 (DOC)']: 16625 tokens -> 2 pieces [9855, 6839]
- ['29_Numeric_Operations__P01 (DOC)']: 17805 tokens -> 2 pieces [15849, 2025]
- ['03_ABAP_SQL__P01 (DOC)']: 17305 tokens -> 2 pieces [14168, 3198]
- ['ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS)']: 18308 tokens -> 2 pieces [15842, 2531]
- ['14_ABAP_Unit_Tests__P04 (DOC)']: 17633 tokens -> 2 pieces [15834, 1866]
- ['24_Builtin_Functions (DOC)']: 16851 tokens -> 2 pieces [12541, 4365]
- ['ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)']: 19740 tokens -> 2 pieces [15825, 3964]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS)']: 17355 tokens -> 2 pieces [15832, 1584]
- ['ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS)']: 20446 tokens -> 2 pieces [15454, 5071]
- ['ZCL_DEMO_ABAP_STRING_PROC (CLAS)']: 20311 tokens -> 2 pieces [15590, 4797]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS)']: 21483 tokens -> 2 pieces [15815, 5730]
- ['ZCL_DEMO_ABAP_DATE_TIME (CLAS)']: 18634 tokens -> 2 pieces [15577, 3138]
- ['ZCL_DEMO_ABAP_XML_JSON (CLAS)']: 16825 tokens -> 2 pieces [15753, 1154]
- ['ZCL_DEMO_ABAP_SQL (CLAS)']: 18283 tokens -> 2 pieces [12526, 5831]
- ['ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS)']: 17309 tokens -> 2 pieces [2124, 15283]
- ['ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS)']: 20979 tokens -> 2 pieces [15831, 5210]
- ['16_Data_Types_and_Objects__P02 (DOC)']: 17891 tokens -> 2 pieces [14421, 3541]

924
train/data/stats.json
View File

@@ -3,348 +3,220 @@
"seed": 20261003,
"max_len": 16384,
"estimate_tokens": 2636112,
"real_tokens": 2806501,
"ratio_real_to_estimate": 1.065,
"all": {
"docs": 370,
"tokens": 2806501,
"min": 49,
"p50": 5195,
"p90": 16923,
"p99": 21483,
"max": 31552
},
"kept": {
"docs": 325,
"tokens": 1966365,
"min": 49,
"p50": 4027,
"p90": 14998,
"p99": 16217,
"max": 16273
},
"train": {
"docs": 309,
"tokens": 1886475,
"min": 49,
"p50": 4403,
"p90": 14998,
"p99": 16217,
"max": 16273
},
"valid": {
"docs": 16,
"tokens": 79890,
"min": 130,
"p50": 3392,
"p90": 16027,
"p99": 16129,
"max": 16129
},
"removed": [
"records": 370,
"tokens_before_dedup": 2806501,
"tokens_after_dedup": 2699110,
"records_after_dedup": 351,
"tokens_after_split": 2701996,
"objects_with_versions": 39,
"families": 269,
"versions_kept": 319,
"versions_removed": 16,
"strict_rule_would_remove_more": 8,
"removed_versions": [
{
"objects": [
"ZCL_DEMO_ABAP_OBJECTS (CLAS)"
],
"tokens": 17042
"object": "ZBP_DEMO_ABAP_RAP_RO_U (CLAS)",
"version": "v758",
"newest": "main",
"diff": 0.015,
"tokens": 10696
},
{
"objects": [
"ZCL_DEMO_ABAP_STRUCTURES (CLAS)"
],
"tokens": 16628
"object": "ZDEMO_ABAP_RAP_RO_M (BDEF)",
"version": "v756",
"newest": "main",
"diff": 0.037,
"tokens": 259
},
{
"objects": [
"32_Performance_Notes__P03 (DOC)"
],
"tokens": 16742
"object": "ZDEMO_ABAP_RAP_RO_U (BDEF)",
"version": "v756",
"newest": "main",
"diff": 0.043,
"tokens": 236
},
{
"objects": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)"
],
"tokens": 19117
"object": "ZDEMO_ABAP_CDS_VE_ASSOC_E (DDLS)",
"version": "v757",
"newest": "main",
"diff": 0.026,
"tokens": 542
},
{
"objects": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)"
],
"tokens": 18883
"object": "ZDEMO_ABAP_CDS_VE_JOINS (DDLS)",
"version": "v757",
"newest": "main",
"diff": 0.009,
"tokens": 1827
},
{
"objects": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS)"
],
"tokens": 19801
"object": "ZDEMO_ABAP_CDS_VE_AGG_EXP (DDLS)",
"version": "v757",
"newest": "main",
"diff": 0.019,
"tokens": 804
},
{
"objects": [
"23_Date_and_Time (DOC)"
],
"tokens": 20347
"object": "ZDEMO_ABAP_CDS_VE_SEL (DDLS)",
"version": "v757",
"newest": "main",
"diff": 0.029,
"tokens": 3273
},
{
"objects": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS)"
],
"tokens": 17477
"object": "ZDEMO_ABAP_CDS_VE_ASSOC (DDLS)",
"version": "v757",
"newest": "main",
"diff": 0.008,
"tokens": 2451
},
{
"objects": [
"07_String_Processing__P02 (DOC)"
],
"tokens": 17916
"object": "ZDEMO_ABAP_FLI_VE (DDLS)",
"version": "v816",
"newest": "main",
"diff": 0.0,
"tokens": 168
},
{
"objects": [
"ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS)"
],
"tokens": 17368
"object": "ZCL_DEMO_ABAP_DTYPE_DOBJ (CLAS)",
"version": "v816",
"newest": "main",
"diff": 0.009,
"tokens": 27407
},
{
"objects": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS)"
],
"tokens": 31552
"object": "ZCL_DEMO_ABAP_DISPLAY (CLAS)",
"version": "v755",
"newest": "v757",
"diff": 0.02,
"tokens": 1677
},
{
"objects": [
"ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS)"
],
"tokens": 19499
"object": "ZCL_DEMO_ABAP_UNIT_TEST (CLAS)",
"version": "v758",
"newest": "main",
"diff": 0.034,
"tokens": 19799
},
{
"objects": [
"04_ABAP_Object_Orientation__P06 (DOC)"
],
"tokens": 16694
"object": "ZDEMO_ABAP_DYNPRO (PROG)",
"version": "v757",
"newest": "v816",
"diff": 0.001,
"tokens": 8026
},
{
"objects": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS)"
],
"tokens": 18368
"object": "ZBP_DEMO_ABAP_RAP_DRAFT_M (CLAS)",
"version": "v757",
"newest": "v758",
"diff": 0.005,
"tokens": 2474
},
{
"objects": [
"06_Dynamic_Programming__P07 (DOC)"
],
"tokens": 23994
"object": "ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)",
"version": "v816",
"newest": "main",
"diff": 0.003,
"tokens": 20484
},
{
"objects": [
"28_Regular_Expressions (DOC)"
],
"tokens": 17900
},
{
"objects": [
"06_Dynamic_Programming__P06 (DOC)"
],
"tokens": 16923
},
{
"objects": [
"ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS)"
],
"tokens": 19769
},
{
"objects": [
"05_Constructor_Expressions__P01 (DOC)"
],
"tokens": 17349
},
{
"objects": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS)"
],
"tokens": 17237
},
{
"objects": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS)"
],
"tokens": 18069
},
{
"objects": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
],
"tokens": 17923
},
{
"objects": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
],
"tokens": 16924
},
{
"objects": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
],
"tokens": 16770
},
{
"objects": [
"14_ABAP_Unit_Tests__P03 (DOC)"
],
"tokens": 18549
},
{
"objects": [
"ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS)"
],
"tokens": 22096
},
{
"objects": [
"22_Released_ABAP_Classes__P01 (DOC)"
],
"tokens": 19022
},
{
"objects": [
"21_XML_JSON__P01 (DOC)"
],
"tokens": 16625
},
{
"objects": [
"29_Numeric_Operations__P01 (DOC)"
],
"tokens": 17805
},
{
"objects": [
"03_ABAP_SQL__P01 (DOC)"
],
"tokens": 17305
},
{
"objects": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS)"
],
"tokens": 18308
},
{
"objects": [
"14_ABAP_Unit_Tests__P04 (DOC)"
],
"tokens": 17633
},
{
"objects": [
"24_Builtin_Functions (DOC)"
],
"tokens": 16851
},
{
"objects": [
"ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)"
],
"tokens": 19740
},
{
"objects": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS)"
],
"tokens": 17355
},
{
"objects": [
"ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS)"
],
"tokens": 20446
},
{
"objects": [
"ZCL_DEMO_ABAP_STRING_PROC (CLAS)"
],
"tokens": 20311
},
{
"objects": [
"ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS)"
],
"tokens": 16394
},
{
"objects": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS)"
],
"tokens": 21483
},
{
"objects": [
"ZCL_DEMO_ABAP_DATE_TIME (CLAS)"
],
"tokens": 18634
},
{
"objects": [
"ZCL_DEMO_ABAP_XML_JSON (CLAS)"
],
"tokens": 16825
},
{
"objects": [
"ZCL_DEMO_ABAP_SQL (CLAS)"
],
"tokens": 18283
},
{
"objects": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS)"
],
"tokens": 17309
},
{
"objects": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS)"
],
"tokens": 20979
},
{
"objects": [
"16_Data_Types_and_Objects__P02 (DOC)"
],
"tokens": 17891
"object": "ZDEMO_ABAP_ALV (PROG)",
"version": "v757",
"newest": "v816",
"diff": 0.002,
"tokens": 7268
}
],
"split_docs": 43,
"hard_line_splits": 17,
"pieces_over_limit": 0,
"train": {
"docs": 374,
"tokens": 2567180,
"min": 49,
"p50": 4904,
"p90": 15638,
"p99": 16177,
"max": 16252
},
"valid": {
"docs": 20,
"tokens": 134816,
"min": 92,
"p50": 7589,
"p90": 15317,
"p99": 16273,
"max": 16273
},
"all": {
"docs": 394,
"tokens": 2701996,
"min": 49,
"p50": 4904,
"p90": 15608,
"p99": 16217,
"max": 16273
},
"share_before_dedup": {
"CLAS": [
1378559,
49.1
],
"other": [
126244,
4.5
],
"DOC": [
1301698,
46.4
]
},
"share_final": {
"CLAS": [
1297878,
48.0
],
"other": [
101390,
3.8
],
"DOC": [
1302728,
48.2
]
},
"by_type": {
"CLAS": {
"docs": 147,
"tokens": 827969
"docs": 194,
"tokens": 1297878
},
"mixed(4)": {
"docs": 6,
"tokens": 28405
},
"DOC": {
"docs": 106,
"tokens": 1012152
"docs": 138,
"tokens": 1302728
},
"INTF": {
"docs": 7,
"tokens": 1792
},
"BDEF": {
"docs": 6,
"tokens": 1564
"docs": 4,
"tokens": 1069
},
"DDLS": {
"docs": 27,
"tokens": 20614
"docs": 21,
"tokens": 11549
},
"mixed(3)": {
"docs": 2,
"tokens": 6667
},
"PROG": {
"docs": 13,
"tokens": 55365
"docs": 11,
"tokens": 40071
},
"DDLX": {
"docs": 6,
@@ -363,8 +235,480 @@
"tokens": 2026
}
},
"source": [
"abap-cheat-sheets"
"split_info": [
{
"object": [
"ZCL_DEMO_ABAP_OBJECTS (CLAS)"
],
"tokens": 17042,
"pieces": 2,
"piece_tokens": [
14777,
2342
]
},
{
"object": [
"ZCL_DEMO_ABAP_STRUCTURES (CLAS)"
],
"tokens": 16628,
"pieces": 2,
"piece_tokens": [
14200,
2503
]
},
{
"object": [
"32_Performance_Notes__P03 (DOC)"
],
"tokens": 16742,
"pieces": 2,
"piece_tokens": [
15846,
961
]
},
{
"object": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)"
],
"tokens": 19117,
"pieces": 2,
"piece_tokens": [
14425,
4770
]
},
{
"object": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS)"
],
"tokens": 18883,
"pieces": 2,
"piece_tokens": [
14607,
4324
]
},
{
"object": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS)"
],
"tokens": 19801,
"pieces": 2,
"piece_tokens": [
15826,
4036
]
},
{
"object": [
"23_Date_and_Time (DOC)"
],
"tokens": 20347,
"pieces": 2,
"piece_tokens": [
10310,
10098
]
},
{
"object": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS)"
],
"tokens": 17477,
"pieces": 2,
"piece_tokens": [
2125,
15451
]
},
{
"object": [
"07_String_Processing__P02 (DOC)"
],
"tokens": 17916,
"pieces": 2,
"piece_tokens": [
11096,
6879
]
},
{
"object": [
"ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS)"
],
"tokens": 17368,
"pieces": 2,
"piece_tokens": [
14031,
3415
]
},
{
"object": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS)"
],
"tokens": 31552,
"pieces": 2,
"piece_tokens": [
15833,
15780
]
},
{
"object": [
"ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS)"
],
"tokens": 19499,
"pieces": 2,
"piece_tokens": [
15831,
3729
]
},
{
"object": [
"04_ABAP_Object_Orientation__P06 (DOC)"
],
"tokens": 16694,
"pieces": 2,
"piece_tokens": [
9045,
7716
]
},
{
"object": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS)"
],
"tokens": 18368,
"pieces": 2,
"piece_tokens": [
15833,
2600
]
},
{
"object": [
"06_Dynamic_Programming__P07 (DOC)"
],
"tokens": 23994,
"pieces": 2,
"piece_tokens": [
15836,
8221
]
},
{
"object": [
"28_Regular_Expressions (DOC)"
],
"tokens": 17900,
"pieces": 2,
"piece_tokens": [
15031,
2930
]
},
{
"object": [
"06_Dynamic_Programming__P06 (DOC)"
],
"tokens": 16923,
"pieces": 2,
"piece_tokens": [
15842,
1144
]
},
{
"object": [
"05_Constructor_Expressions__P01 (DOC)"
],
"tokens": 17349,
"pieces": 2,
"piece_tokens": [
15147,
2265
]
},
{
"object": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS)"
],
"tokens": 17237,
"pieces": 2,
"piece_tokens": [
15833,
1469
]
},
{
"object": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS)"
],
"tokens": 18069,
"pieces": 2,
"piece_tokens": [
15842,
2292
]
},
{
"object": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
],
"tokens": 17923,
"pieces": 2,
"piece_tokens": [
15329,
2672
]
},
{
"object": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
],
"tokens": 16924,
"pieces": 2,
"piece_tokens": [
15749,
1223
]
},
{
"object": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS)"
],
"tokens": 16770,
"pieces": 2,
"piece_tokens": [
14210,
2608
]
},
{
"object": [
"14_ABAP_Unit_Tests__P03 (DOC)"
],
"tokens": 18549,
"pieces": 2,
"piece_tokens": [
15834,
2782
]
},
{
"object": [
"ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS)"
],
"tokens": 22096,
"pieces": 2,
"piece_tokens": [
15814,
6342
]
},
{
"object": [
"22_Released_ABAP_Classes__P01 (DOC)"
],
"tokens": 19022,
"pieces": 2,
"piece_tokens": [
14661,
4430
]
},
{
"object": [
"21_XML_JSON__P01 (DOC)"
],
"tokens": 16625,
"pieces": 2,
"piece_tokens": [
9855,
6839
]
},
{
"object": [
"29_Numeric_Operations__P01 (DOC)"
],
"tokens": 17805,
"pieces": 2,
"piece_tokens": [
15849,
2025
]
},
{
"object": [
"03_ABAP_SQL__P01 (DOC)"
],
"tokens": 17305,
"pieces": 2,
"piece_tokens": [
14168,
3198
]
},
{
"object": [
"ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS)"
],
"tokens": 18308,
"pieces": 2,
"piece_tokens": [
15842,
2531
]
},
{
"object": [
"14_ABAP_Unit_Tests__P04 (DOC)"
],
"tokens": 17633,
"pieces": 2,
"piece_tokens": [
15834,
1866
]
},
{
"object": [
"24_Builtin_Functions (DOC)"
],
"tokens": 16851,
"pieces": 2,
"piece_tokens": [
12541,
4365
]
},
{
"object": [
"ZCL_DEMO_ABAP_NUMERIC_OP (CLAS)"
],
"tokens": 19740,
"pieces": 2,
"piece_tokens": [
15825,
3964
]
},
{
"object": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS)"
],
"tokens": 17355,
"pieces": 2,
"piece_tokens": [
15832,
1584
]
},
{
"object": [
"ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS)"
],
"tokens": 20446,
"pieces": 2,
"piece_tokens": [
15454,
5071
]
},
{
"object": [
"ZCL_DEMO_ABAP_STRING_PROC (CLAS)"
],
"tokens": 20311,
"pieces": 2,
"piece_tokens": [
15590,
4797
]
},
{
"object": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS)"
],
"tokens": 21483,
"pieces": 2,
"piece_tokens": [
15815,
5730
]
},
{
"object": [
"ZCL_DEMO_ABAP_DATE_TIME (CLAS)"
],
"tokens": 18634,
"pieces": 2,
"piece_tokens": [
15577,
3138
]
},
{
"object": [
"ZCL_DEMO_ABAP_XML_JSON (CLAS)"
],
"tokens": 16825,
"pieces": 2,
"piece_tokens": [
15753,
1154
]
},
{
"object": [
"ZCL_DEMO_ABAP_SQL (CLAS)"
],
"tokens": 18283,
"pieces": 2,
"piece_tokens": [
12526,
5831
]
},
{
"object": [
"ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS)"
],
"tokens": 17309,
"pieces": 2,
"piece_tokens": [
2124,
15283
]
},
{
"object": [
"ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS)"
],
"tokens": 20979,
"pieces": 2,
"piece_tokens": [
15831,
5210
]
},
{
"object": [
"16_Data_Types_and_Objects__P02 (DOC)"
],
"tokens": 17891,
"pieces": 2,
"piece_tokens": [
14421,
3541
]
}
],
"iterations_2_epochs": 618
"iterations_2_epochs": 748
}