erhan/abap-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Object type mix: INTF, TABL, STRU, MSAG, exception tasks (harness G2, mutants, generator notes), balanced generator, kind-deficit job order, dashboard mix card

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Kral
2026-10-05 18:52:29 +02:00
parent 454db6626c
commit b119f1afac
39 changed files with 2329 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

75
harness/generator.py
View File

@@ -24,7 +24,8 @@ ABAPLINT = os.path.join(ROOT, "node_modules", ".bin", "abaplint")
MAX_OUT_TOKENS = 80000
RUNS_PER_ATTEMPT = 10
ABAPLINT_VERSIONS = {"v702", "v740sp05", "v750", "v758"}
EXAMPLE_FOR = {"CLAS": "T01", "INTF": "T01", "FUNC": "T13", "PROG": "T14", "DDLS": "T15", "TABL": "T15"}
EXAMPLE_FOR = {"CLAS": "T01", "INTF": "T01", "FUNC": "T13", "PROG": "T14", "DDLS": "T15", "TABL": "T15",
"STRU": "T15", "MSAG": "T13"}
CATEGORIES = {
"A": "pure logic in a new class (language, OO design, Clean ABAP)",
"B": "database access (ABAP SQL, CDS) with test doubles",
@@ -50,6 +51,57 @@ CATEGORIES = {
"hidden_tests is []. Do not write the gap in Open questions (write \"None.\")",
}
# Extra format notes for object types that the main prompt does not describe (training mode, 2026-10-05).
# The example bundle is of another type; these notes say how this type differs.
TYPE_NOTES = {
"INTF": """The model must create an INTERFACE (type INTF) as the main contract object: the methods, types and constants
named in the spec. Contract entry: {"type": "INTF", "name": "{{P}}IF_X"}. Reference: the interface source
("reference/x.intf.abap") and ONE global own-test class (type CLAS, source with a global class DEFINITION FOR TESTING;
an interface has no test include). Hidden tests: one global test class with a local class that IMPLEMENTS the
interface (it only compiles when every signature matches), calls the methods through the interface, reads the
constants (values), and checks type lengths with RTTI (cl_abap_typedescr=>describe_by_name). The business rules give the
exact constant values, parameter names and types. No other object is needed.""",
"TABL": """The model must create a transparent DATABASE TABLE (type TABL) as the main contract object. The table name has at
most 16 characters including the prefix: use a short name, for example {{P}}ORD (7 characters after the prefix at most).
Source is DDL: annotations @EndUserText.label, @AbapCatalog.enhancement.category : #NOT_EXTENSIBLE,
@AbapCatalog.tableCategory : #TRANSPARENT, @AbapCatalog.deliveryClass : #A, @AbapCatalog.dataMaintenance : #RESTRICTED,
then "define table {{p}}ord { key client : abap.clnt not null; key item_id : abap.char(10) not null; ... }".
Contract entry: {"type": "TABL", "name": "{{P}}ORD", "fields": ["client", "item_id", ...]} (all fields, lower case).
The spec (Business rules) lists every field with its data type and length, the key fields, and not-null fields. Use
only built-in types (abap.char, abap.numc, abap.dec, abap.int4, abap.dats). Avoid abap.curr and abap.quan; if the spec
needs an amount or a quantity, the reference field needs a reference annotation in the form 'tablename.fieldname'
(for example @Semantics.amount.currencyCode : '{{p}}ord.currency_code' on the amount and a field currency_code :
abap.cuky in the same table; @Semantics.quantity.unitOfMeasure : '{{p}}ord.unit' with unit : abap.unit(3)); a
reference without the table name fails with "Annotation with reference to unit code ... is uncomplete". Reference: the DDL file and ONE global own-test class
(type CLAS). Hidden tests (one global class): RTTI on the table: cast cl_abap_typedescr=>describe_by_name( '{{P}}ORD' ) to
cl_abap_structdescr and check get_ddic_field_list( ) (field names, key flags, lengths, decimals, types), and a
cl_osql_test_environment test (create( i_dependency_list = VALUE #( ( '{{P}}ORD' ) ) ), insert, select back; a second
INSERT with the same key gives sy-subrc = 4). No seed is needed.""",
"STRU": """The model must create a DDIC STRUCTURE (type STRU) as the main contract object. DDL source: annotations
@EndUserText.label and @AbapCatalog.enhancement.category : #NOT_EXTENSIBLE, then
"define structure {{p}}name { code : abap.char(4); amount : abap.dec(9,2); ... }" (include another structure with
"include {{p}}other;" when the spec asks). Contract entry: {"type": "STRU", "name": "{{P}}NAME", "fields": [...]}
(the field names in lower case). The Business rules list every field with data type and length. Reference: the DDL file
and ONE global own-test class (type CLAS). Hidden tests (one global class): RTTI: cl_abap_typedescr=>describe_by_name(
'{{P}}NAME' ) cast to cl_abap_structdescr; check components (names, length, decimals, type kind) and the order.""",
"MSAG": """The model must create a MESSAGE CLASS (type MSAG) as the main contract object. The name has at most 20
characters including the prefix. A message class has NO source file. In task.json the reference entry has no "file":
{"type": "MSAG", "name": "{{P}}MSG", "description": "...", "messages": [{"msgno": "001", "text": "Order &1 is blocked"}, ...]}
and the contract entry is {"type": "MSAG", "name": "{{P}}MSG", "messages": [{"msgno": "001"}, ...]} (the numbers that
must exist). The Business rules give for each message the number, the exact text with the placeholders &1 to &4 (at
most 72 characters), and when it is used (error, warning, information). Reference: the message class entry and ONE global
own-test class (type CLAS) that reads the messages. Hidden tests (one global class): for each message use
MESSAGE ID '{{P}}MSG' TYPE 'E' NUMBER '001' WITH 'A' 'B' INTO DATA(lv_text) and assert the final text; also check one
placeholder substitution. The model may also be asked for an exception class that uses the message class (then both
are contract entries and the exception class is a reference file).""",
"EXC": """The main contract object is a class-based EXCEPTION class (CLAS, name {{P}}CX_...) that inherits from
CX_STATIC_CHECK, CX_DYNAMIC_CHECK or CX_NO_CHECK. The spec asks for constants for the error cases, attributes with
context data, a constructor with these parameters, and a message text (through IF_T100_DYN_MSG / IF_T100_MESSAGE with
a message class that is a seed object, or through a redefined get_text). Hidden tests raise the exception from a small
seed class, catch it and check the attributes and the text (get_text( )). Reference: the exception class with its
testclasses_file or ONE global own-test class (an exception class has few methods).""",
}
SYSTEM = """You write evaluation tasks for an ABAP developer model. Each task is a bundle of files.
The model gets only spec.md and works on an SAP ABAP Platform 2025 system (SAP_BASIS 816, client 001)
through ADT tools. The harness installs the seed objects, runs the model, then checks the result with
@@ -220,7 +272,7 @@ def lint_files(b):
m = DDL_KIND.search(b.get("files", {}).get(o.get("file", ""), ""))
if not m:
continue
want = {"table": "TABL", "structure": "TABL", "view": "DDLS"}[m.group(1).lower()]
want = {"table": "TABL", "structure": "STRU", "view": "DDLS"}[m.group(1).lower()]
if o.get("type") != want:
errs.append(f"{k}: {o.get('name')} has type {o.get('type')}, but its source is "
f"'define {m.group(1)}'; use type {want}")
@@ -294,9 +346,24 @@ def check_bundle(b):
errs.append(f"{k}: file {o[fk]} missing")
if not o.get("name", "").startswith("{{P}}"):
errs.append(f"{k}: name {o.get('name')} does not start with {{{{P}}}}")
limit = 16 if o.get("type") == "TABL" else 26 if o.get("type") == "FUGR" else 30
limit = {"TABL": 16, "FUGR": 26, "MSAG": 20}.get(o.get("type"), 30)
if len(o.get("name", "").replace("{{P}}", "Z0000000_")) > limit:
errs.append(f"{k}: name {o.get('name')} too long (max {limit})")
for k in ("seed", "reference"):
for o in t.get(k, []):
if o.get("type") == "MSAG":
msgs = o.get("messages") or []
if k == "reference" and not msgs:
errs.append(f"reference message class {o.get('name')} has no 'messages' list")
for m in msgs:
if not re.fullmatch(r"\d{1,3}", str(m.get("msgno", ""))) or not m.get("text") or len(m["text"]) > 72:
errs.append(f"message class {o.get('name')}: message {m.get('msgno')} needs a 1 to 3 digit "
"msgno and a text of at most 72 characters")
for c in t.get("contract", []):
if c.get("type") in ("TABL", "STRU") and not c.get("fields"):
errs.append(f"contract {c.get('name')}: a {c['type']} entry needs the list 'fields'")
if c.get("type") == "MSAG" and not c.get("messages"):
errs.append(f"contract {c.get('name')}: a MSAG entry needs the list 'messages'")
if not t.get("hidden_tests") and not stop:
errs.append("no hidden test class")
return errs
@@ -399,9 +466,11 @@ def generate(task_id, pool, object_type, category, difficulty, model, base_url,
pool_root = os.path.join(ROOT, "tasks_gen", pool)
task_dir = os.path.join(pool_root, task_id)
example = bundle_of(os.path.join(ROOT, "tasks", EXAMPLE_FOR[object_type]))
note_key = "EXC" if (object_type == "CLAS" and topic and "exception class" in topic.lower()) else object_type
ask = (f"Write one new task.\nObject type of the main contract object: {object_type}.\n"
f"Skill category {category}: {CATEGORIES[category]}.\nDifficulty {difficulty} of 3.\n"
+ (f"Topic idea: {topic}\n" if topic else "Choose a new, realistic business topic.\n")
+ (("Format notes for this object type:\n" + TYPE_NOTES[note_key] + "\n") if note_key in TYPE_NOTES else "")
+ "Here is an example bundle of a different task (same format):\n" + json.dumps(example))
messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": ask}]
log = {"id": task_id, "pool": pool, "object_type": object_type, "category": category, "attempts": []}