erhan/abap-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Mutation in generator; category H (stop) scoring and judge; category K variants and tool schema variant; evalset plan

Browse Source 
- mutation.py: negation mutants, skip WHILE/DO blocks (endless loop blocked RFC ~8 min)
- generator: accept only after mutation check; survivors go back as repair feedback
- runner/judge.py: stop tasks scored 100/30/0; gap by keywords, else judge model
- proxy: tool schema variant generic_v0 (draft); generator make_k_variant
- evalset.py: 88 slots for step F (A-I, H, K)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kral
2026-10-03 05:37:20 +02:00
parent 32f03fcad4
commit 0ba25faeea
20 changed files with 1303 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
tasks_gen/eval/G0003/mutation.json Normal file
View File

@@ -0,0 +1,68 @@
{
"task": "G0003",
"mutants": [
{
"object": "{{P}}COURSE_BOOKING",
"mutant": "line 52: < -> >= (rel)",
"status": "killed",
"hidden": "9/11",
"failed_tests": [
"BOOKABLE_EXACT_FREE",
"NOT_BOOKABLE_ZERO_SEATS"
]
},
{
"object": "{{P}}COURSE_BOOKING",
"mutant": "line 53: abap_false -> abap_true (bool)",
"status": "killed",
"hidden": "10/11",
"failed_tests": [
"NOT_BOOKABLE_ZERO_SEATS"
]
},
{
"object": "{{P}}COURSE_BOOKING",
"mutant": "line 69: < -> >= (rel)",
"status": "killed",
"hidden": "7/11",
"failed_tests": [
"LIST_EMPTY_WITHOUT_MATCH",
"LIST_EXCLUDES_FULL",
"LIST_FILTER_CAT_SEATS",
"LIST_SORTED_BY_COURSE"
]
},
{
"object": "{{P}}COURSE_BOOKING",
"mutant": "line 84: = -> <> (eq)",
"status": "killed",
"hidden": "6/11",
"failed_tests": [
"BOOKABLE_EXACT_FREE",
"FREE_SEATS_NEVER_NEG",
"FREE_SEATS_OPEN_COURSE",
"FREE_SEATS_UNKNOWN",
"NOT_BOOKABLE_UNKNOWN"
]
},
{
"object": "{{P}}COURSE_BOOKING",
"mutant": "line 89: - -> + (arith)",
"status": "killed",
"hidden": "4/11",
"failed_tests": [
"FREE_SEATS_NEVER_NEG",
"FREE_SEATS_OPEN_COURSE",
"LIST_EMPTY_WITHOUT_MATCH",
"LIST_EXCLUDES_FULL",
"LIST_FILTER_CAT_SEATS",
"LIST_SORTED_BY_COURSE",
"NOT_BOOKABLE_TOO_MANY"
]
}
],
"valid": 5,
"killed": 5,
"kill_rate": 1.0,
"ok": true
}