Mutation in generator; category H (stop) scoring and judge; category K variants and tool schema variant; evalset plan
- mutation.py: negation mutants, skip WHILE/DO blocks (endless loop blocked RFC ~8 min) - generator: accept only after mutation check; survivors go back as repair feedback - runner/judge.py: stop tasks scored 100/30/0; gap by keywords, else judge model - proxy: tool schema variant generic_v0 (draft); generator make_k_variant - evalset.py: 88 slots for step F (A-I, H, K) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
59
tasks_gen/eval/G0002/mutation.json
Normal file
59
tasks_gen/eval/G0002/mutation.json
Normal file
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"task": "G0002",
|
||||
"mutants": [
|
||||
{
|
||||
"object": "{{P}}DELIVERY_SCHEDULE",
|
||||
"mutant": "line 68: <> -> = (rel)",
|
||||
"status": "killed",
|
||||
"hidden": "9/12",
|
||||
"failed_tests": [
|
||||
"ADD_DAYS_BACKWARD",
|
||||
"ADD_DAYS_FORWARD",
|
||||
"ADD_FORWARD_SKIPS_HOLIDAY"
|
||||
]
|
||||
},
|
||||
{
|
||||
"object": "{{P}}DELIVERY_SCHEDULE",
|
||||
"mutant": "line 72: - -> + (arith)",
|
||||
"status": "invalid",
|
||||
"hidden": "0/0",
|
||||
"failed_tests": []
|
||||
},
|
||||
{
|
||||
"object": "{{P}}DELIVERY_SCHEDULE",
|
||||
"mutant": "line 81: = -> <> (eq)",
|
||||
"status": "invalid",
|
||||
"hidden": "0/0",
|
||||
"failed_tests": []
|
||||
},
|
||||
{
|
||||
"object": "{{P}}DELIVERY_SCHEDULE",
|
||||
"mutant": "line 91: 5 -> 6 (const)",
|
||||
"status": "killed",
|
||||
"hidden": "7/12",
|
||||
"failed_tests": [
|
||||
"ADD_DAYS_BACKWARD",
|
||||
"ADD_DAYS_FORWARD",
|
||||
"ADD_FORWARD_SKIPS_HOLIDAY",
|
||||
"NEXT_DAY_FROM_WEEKEND",
|
||||
"WEEKEND_IS_NOT_WORKING"
|
||||
]
|
||||
},
|
||||
{
|
||||
"object": "{{P}}DELIVERY_SCHEDULE",
|
||||
"mutant": "line 98: abap_true -> abap_false (bool)",
|
||||
"status": "killed",
|
||||
"hidden": "8/12",
|
||||
"failed_tests": [
|
||||
"ADD_FORWARD_SKIPS_HOLIDAY",
|
||||
"COUNT_WORKING_DAYS_INCLUSIVE",
|
||||
"HOLIDAY_IS_NOT_WORKING",
|
||||
"NEXT_DAY_SKIPS_HOLIDAYS"
|
||||
]
|
||||
}
|
||||
],
|
||||
"valid": 3,
|
||||
"killed": 3,
|
||||
"kill_rate": 1.0,
|
||||
"ok": true
|
||||
}
|
||||
Reference in New Issue
Block a user