diff --git a/docs/own-test-mutation.md b/docs/own-test-mutation.md new file mode 100644 index 0000000..b6688a4 --- /dev/null +++ b/docs/own-test-mutation.md @@ -0,0 +1,20 @@ +# Own-test mutation score of the accepted trajectories (item D, metadata only) + +Module `harness/owntests.py`, results `runs/traj//own_test_mutation.json` (not in git), report by `train/own_test_report.py`. +The model's own unit tests (testclasses include or global test classes) are run against the faulty references of the task (`faulty/`, mutants that the hidden tests kill). +The correct reference must pass the model's tests first (otherwise the tests encode model specific behavior). score = killed / (killed + survived). +**Metadata only: the acceptance filter is not changed.** The builder takes the score through `--hook hooks_example:own_test_weight` (field `own_test_mutation`). + +## Result +90 accepted trajectories looked at: {'scored': 63, 'not_supported': 5, 'no_own_tests': 22}. +- Scored: 63; the model's tests also passed on the correct reference: 53 (the others are not reliable: a test that fails on the correct solution kills every mutant). +- Mean score over the reliable ones: 0.99; median 1.00; distribution {'1.0': 50, '>=0.75': 3} (n = 53). +- By kind (mean, n): {CLAS: 1.00 (45), DDLS: 0.90 (5), FUNC: 1.00 (3)} +- `no_own_tests`: 22 trajectories were accepted without any own test (at most 85 points); `no_mutants`: 0; `not_supported` (PROG: tests are inside the program): 5. + +## Weakest (score under 0.75, tests pass on the reference) + + +## Use +Not used for filtering yet. Candidates for a later rule (Kral + Opus decide): drop or down-weight trajectories with a reliable score under 0.5 or with `tests_pass_on_reference` false; +prefer trajectories without own tests last. The hook example shows where the weight goes (`train/hooks_example.py`). diff --git a/docs/stage2-build.md b/docs/stage2-build.md index 553dbd9..2310541 100644 --- a/docs/stage2-build.md +++ b/docs/stage2-build.md @@ -1,4 +1,4 @@ -# Stage 2 training set, build of 2026-10-06 06:15:28 +# Stage 2 training set, build of 2026-10-06 07:36:30 Builder: `train/build_stage2.py` (output `runs/stage2_data/`, data card `README.md`, report `build_report.json`; this page: `train/build_doc.py`). Hook for item D: `train/hooks_example.py`. HF dataset (private): `erhankeseli/abap-stage2-data`. The local Qwen trajectories (series A) are not read.