Mutation in generator; category H (stop) scoring and judge; category K variants and tool schema variant; evalset plan
- mutation.py: negation mutants, skip WHILE/DO blocks (endless loop blocked RFC ~8 min) - generator: accept only after mutation check; survivors go back as repair feedback - runner/judge.py: stop tasks scored 100/30/0; gap by keywords, else judge model - proxy: tool schema variant generic_v0 (draft); generator make_k_variant - evalset.py: 88 slots for step F (A-I, H, K) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -43,6 +43,10 @@ class OracleAgent:
|
||||
name = "oracle"
|
||||
|
||||
def run(self, task, proxy):
|
||||
if task.meta.get("expected_outcome") == "stop": # the right answer is to stop and name the gap
|
||||
report = "Stopped. Gap: " + task.meta.get("expected_gap", {}).get("description", "")
|
||||
proxy.note("final", report)
|
||||
return report
|
||||
for o in task.objects("reference"):
|
||||
ident = {"objectType": o["type"], "objectName": o["name"]}
|
||||
if o.get("functionGroup"):
|
||||
|
||||
Reference in New Issue
Block a user