e083c9ca13
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014aUaQeLnwbb1zTpN7kHeat
99 lines
3.6 KiB
Markdown
99 lines
3.6 KiB
Markdown
# Stage 1 data report
|
|
|
|
Corpus: `/Users/erhankeseli/projects/abap-llm/corpus/corpus.jsonl` (source abap-cheat-sheets, Apache-2.0, see corpus/out/ATTRIBUTION.md).
|
|
Tokenizer: `/Users/erhankeseli/models/Qwen3.8-27B-4bit`. Seed 20261003. max_seq_length 16384.
|
|
Split by family (release versions and parts of one document stay together): 180 families.
|
|
|
|
| | docs | tokens |
|
|
|---|---|---|
|
|
| corpus | 370 | 2806501 (estimate chars/4: 2636112, ratio 1.065) |
|
|
| removed (> 16384) | 45 | 840136 |
|
|
| train | 309 | 1886475 |
|
|
| valid (= test file) | 16 | 79890 |
|
|
|
|
Iterations for 2 epochs (batch 1): 618.
|
|
|
|
## Token distribution per document (real tokenizer)
|
|
|
|
| set | min | p50 | p90 | p99 | max |
|
|
|---|---|---|---|---|---|
|
|
| all | 49 | 5195 | 16923 | 21483 | 31552 |
|
|
| kept | 49 | 4027 | 14998 | 16217 | 16273 |
|
|
| train | 49 | 4403 | 14998 | 16217 | 16273 |
|
|
| valid | 130 | 3392 | 16027 | 16129 | 16129 |
|
|
|
|
| tokens per document up to | docs |
|
|
|---|---|
|
|
| 512 | 48 |
|
|
| 1024 | 25 |
|
|
| 2048 | 40 |
|
|
| 4096 | 51 |
|
|
| 8192 | 53 |
|
|
| 16384 | 108 |
|
|
| more | 45 |
|
|
|
|
## By object type (kept documents)
|
|
|
|
| types | docs | tokens |
|
|
|---|---|---|
|
|
| DOC | 106 | 1012152 |
|
|
| CLAS | 147 | 827969 |
|
|
| PROG | 13 | 55365 |
|
|
| mixed(4) | 6 | 28405 |
|
|
| DDLS | 27 | 20614 |
|
|
| mixed(3) | 2 | 6667 |
|
|
| DDLX | 6 | 5003 |
|
|
| CLAS+DDLS | 1 | 4563 |
|
|
| FUGR | 2 | 2026 |
|
|
| INTF | 7 | 1792 |
|
|
| BDEF | 6 | 1564 |
|
|
| DCLS | 2 | 245 |
|
|
|
|
## Removed documents
|
|
|
|
- ZCL_DEMO_ABAP_OBJECTS (CLAS): 17042 tokens
|
|
- ZCL_DEMO_ABAP_STRUCTURES (CLAS): 16628 tokens
|
|
- 32_Performance_Notes__P03 (DOC): 16742 tokens
|
|
- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 19117 tokens
|
|
- ZCL_DEMO_ABAP_INTERNAL_TABLES (CLAS): 18883 tokens
|
|
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V755 (CLAS): 19801 tokens
|
|
- 23_Date_and_Time (DOC): 20347 tokens
|
|
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V757 (CLAS): 17477 tokens
|
|
- 07_String_Processing__P02 (DOC): 17916 tokens
|
|
- ZCL_DEMO_ABAP_CONSTRUCTOR_EXPR (CLAS): 17368 tokens
|
|
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V758 (CLAS): 31552 tokens
|
|
- ZCL_DEMO_ABAP_STRING_PROC__V757 (CLAS): 19499 tokens
|
|
- 04_ABAP_Object_Orientation__P06 (DOC): 16694 tokens
|
|
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V758 (CLAS): 18368 tokens
|
|
- 06_Dynamic_Programming__P07 (DOC): 23994 tokens
|
|
- 28_Regular_Expressions (DOC): 17900 tokens
|
|
- 06_Dynamic_Programming__P06 (DOC): 16923 tokens
|
|
- ZCL_DEMO_ABAP_NUMERIC_OP__V816 (CLAS): 19769 tokens
|
|
- 05_Constructor_Expressions__P01 (DOC): 17349 tokens
|
|
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V755 (CLAS): 17237 tokens
|
|
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V756 (CLAS): 18069 tokens
|
|
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 17923 tokens
|
|
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16924 tokens
|
|
- ZCL_DEMO_ABAP_DYNAMIC_PROG (CLAS): 16770 tokens
|
|
- 14_ABAP_Unit_Tests__P03 (DOC): 18549 tokens
|
|
- ZCL_DEMO_ABAP_STRING_PROC__V758 (CLAS): 22096 tokens
|
|
- 22_Released_ABAP_Classes__P01 (DOC): 19022 tokens
|
|
- 21_XML_JSON__P01 (DOC): 16625 tokens
|
|
- 29_Numeric_Operations__P01 (DOC): 17805 tokens
|
|
- 03_ABAP_SQL__P01 (DOC): 17305 tokens
|
|
- ZCL_DEMO_ABAP_DTYPE_DOBJ__V757 (CLAS): 18308 tokens
|
|
- 14_ABAP_Unit_Tests__P04 (DOC): 17633 tokens
|
|
- 24_Builtin_Functions (DOC): 16851 tokens
|
|
- ZCL_DEMO_ABAP_NUMERIC_OP (CLAS): 19740 tokens
|
|
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V758 (CLAS): 17355 tokens
|
|
- ZCL_DEMO_ABAP_BUILTIN_FUNC (CLAS): 20446 tokens
|
|
- ZCL_DEMO_ABAP_STRING_PROC (CLAS): 20311 tokens
|
|
- ZCL_DEMO_ABAP_UNIT_TEST__V758 (CLAS): 16394 tokens
|
|
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V757 (CLAS): 21483 tokens
|
|
- ZCL_DEMO_ABAP_DATE_TIME (CLAS): 18634 tokens
|
|
- ZCL_DEMO_ABAP_XML_JSON (CLAS): 16825 tokens
|
|
- ZCL_DEMO_ABAP_SQL (CLAS): 18283 tokens
|
|
- ZCL_DEMO_ABAP_INTERNAL_TABLES__V755 (CLAS): 17309 tokens
|
|
- ZCL_DEMO_ABAP_DYNAMIC_PROG__V756 (CLAS): 20979 tokens
|
|
- 16_Data_Types_and_Objects__P02 (DOC): 17891 tokens
|