abap-llm/CLAUDE.md

ABAP LLM harness — instructions for Claude Code

0. Working with Kral

• Talk in Turkish. Short answers. Remove words and sentences that add no value.
• English text (specs, prompts, docs for the model): ASD-STE100 Simplified Technical English.
• Ask before an action that costs much (cloud budget) or that changes the A4H system outside harness runs.
• For a series of updates to one document, deliver the result directly as a Markdown file; do not ask each time.
• Record each decision with its date in docs/yol-haritasi.md or docs/faz1-tasarim.md.
• Commit the current changes before a new generation or test batch.
• Kral has no time for a full review. Eval review = automatic gates + empirical filter (2–3 models) + Claude review with a checklist; Kral reviews only flagged tasks + one per category (~10 tasks).

1. Project

• Goal: train an open-weight ABAP model (Apache 2.0). Base model: Apache 2.0 or MIT only.
• Role of the model: technical ABAP consultant. It writes ABAP, knows Clean ABAP and what to use how. No SAP module knowledge; the functional side (spec or the /sapplan planner) gives it.
• Contract of the model: "ABAP task as text + generic ABAP MCP interface". The plan format (plan-agent.md) gives the best result, but it is not mandatory.
• Training data: never Claude output. Eval tasks never go into training data.
• Input: the plan format gives the best result but is not mandatory. The old playbook format is not used.
• Generic ABAP MCP interface (core: search_object, read_source, object_structure, where_used, create_object, write_source, activate, syntax_check, run_unit_tests, run_atc). Small tool name/schema variation in training.
• Category K: incomplete or free-text input + other tool schema.
• Object type mix: CLAS/INTF 35 %, FUNC 15 %, PROG 10 %, DDIC 10 %, CDS 25 %, MSAG + exception 5 %.
• Teacher and task author: DeepSeek V4.1 Flash (Ollama cloud). Budget calendar in docs/yol-haritasi.md.
• Read first: docs/yol-haritasi.md (roadmap, current position) and docs/faz1-tasarim.md (design). When a decision or a result changes, update these files.

2. Environment (this Mac mini)

• A4H: Docker container a4h, HTTP localhost:50000, client 001, SAP_BASIS 816 SP01.
• MCP server (EPOD) runs inside ADT (Eclipse) at 127.0.0.1:3000. Token: .env (MCP_TOKEN). System name A4H, mode write.
• Ollama 127.0.0.1:11434:
  • qwen3.8-27b-32k (local, num_ctx 32k, ~18 GB). Slow: 20–40 min per task.
  • deepseek-v4.1-flash:cloud (Ollama cloud, MIT): teacher and task author. ~0.05–0.08 USD per run.
• Python 3.9 (system), Node; abaplint in node_modules.

3. Rules

• Local model: one request at a time. No time limit (tool-call budget limits a run).
• Do not run local and cloud model runs at the same time: Ollama can queue them together.
• A4H is small: MCP limit is 8 sessions (by design). The server shares ONE RFC connection between sessions; a parallel call gets "[LOCK] Concurrent call detected". mcp_client.py retries this. Keep parallel runs low (max 3). DDIC activation during setup is sensitive to parallel runs.
• Budget: runs/ledger.jsonl (list prices, upper bound; real cost ≈ ledger / 2.7, measured 2026-10-03 20:15: ledger 66.97 vs Ollama monthly usage 24.63 USD (earlier ratio 1.47 was wrong)). .env: BUDGET_LIMIT_USD, BUDGET_CYCLE_START. Ollama usage resets on 12 October 2026, then +60 USD per month. At the limit, stop cloud work.
• Objects: package $TMP only. Prefix Z + run (4 chars base36) + task (3 chars base36) + _ (harness/task.py). Teardown after each run with the ADT deletion API (adt_client.py, credentials in .env). Delete only objects with a run prefix. Clean up probe objects.
• The model never sees delete/teardown. The proxy (proxy.py) has a tool whitelist and hides other runs' objects.
• To wait for a background job, use its PID: while kill -0 <PID> 2>/dev/null; do sleep 60; done. Do NOT use pgrep -f <pattern> in the loop: it also finds the loop's own command line, so the loop never ends. The process name of a module run is Python -m harness.<module> (not python3); pgrep -f "harness.pilot" without a loop gives the PID.
• Start long jobs with nohup ... & from a separate line after cd (a cd ... && nohup ... & chain runs in a subshell; the commands after it do not see the new directory).

4. Layout

• harness/mcp_client.py MCP client (retry on 404 and on LOCK).
• harness/adt_client.py ADT deletion API.
• harness/proxy.py tool whitelist, budget, prefix filter, trajectory log.
• harness/agents.py oracle, null, llm (OpenAI-compatible; retries; ledger).
• harness/runner.py setup → agent → gates G1–G6 → hidden tests → own tests → ATC → abaplint → score → teardown.
• harness/generator.py task generator (cloud model writes a bundle; validation oracle = 100, null = 0; max 3 repairs).
• harness/pilot.py pilot list (20 tasks G0002–G0021).
• harness/ledger.py cost ledger and budget guard.
• harness/cli.py run, rescore, teardown, teardown-all, cleanup-list.
• tasks/ hand-written tasks T01 (CLAS), T13 (FUNC), T14 (PROG + ALV), T15 (CDS). Oracle 100, null 0 for all.
• tasks_gen/eval/, tasks_gen/train/ generated tasks (separate pools).
• runs/ run results (git ignores it). runs/_archive_v1 old runs.

5. Commands

python3 -m harness.cli run T01 --agent oracle|null|llm [--model NAME] --run N
python3 -m harness.cli rescore runs/<run_dir>
python3 -m harness.cli cleanup-list <PREFIX> runs/<dir> && python3 -m harness.cli teardown runs/<dir>
python3 -m harness.generator --id G0100 --pool eval --object-type CLAS --category C --run-base 2000
python3 -m harness.pilot 2            # pilot list; `python3 -m harness.pilot 22 hard 1400` hard batch
python3 -m harness.mutation G0002 --pool eval --run-base 5000 [--keep]
python3 -c "from harness.ledger import spent; print(spent())"

6. Current state (2026-10-02)

• Done: harness on Mac mini; test include creation in the server; teardown; FUNC/PROG/DDLS support; task generator; ledger and budget guard.
• Results: T01 oracle 100, Qwen 27B 41.7 (old run 103), DeepSeek 98.5. T13 DeepSeek 85, T14 DeepSeek 100.
• Pilot (G0002–G0021): 16/20 accepted, 0.16 USD per accepted task (docs/faz1-tasarim.md 11e). G0020 and G0013 fixed in review and revalidated → 18/20.
• Hard batch (G0022–G0026, 2026-10-03): 5/5 accepted (G0022 after harness fixes), 0.097 USD per accepted task, first-attempt 1/5 (docs/faz1-tasarim.md 11f).
• Generator: static checks, local abaplint parser check, dependency order of seed/reference, budget floor, max_tokens 80k. Runner: G2 for CDS with parameters, G6 ignores unknown standard superclasses.
• Step D done: harness/mutation.py, part of generator acceptance. All 26 tasks pass (docs/faz1-tasarim.md 11g). Killed mutants are kept in <task>/faulty/.
• Step 5 code: H stop scoring (judge.py), K variants (make_k_variant), proxy tool schema generic_v0 (draft). Not yet run on SAP.
• Step F: harness/evalset.py (88 slots G0100–G0187). Not started.

6a. Base model decision (2026-10-04, revised)

• Base model: Qwen 3.8 27B (Qwen/Qwen3.8-27B, Apache 2.0; weights mlx-community/Qwen3.8-27B-4bit, local ~/models/Qwen3.8-27B-4bit). Kral decision, same day, after the Devstral baseline.
• Earlier the same day Qwen was dropped (no repair after activation errors, loops, empty responses at the thinking limit) and Devstral Small 2 (24B) was the candidate. Kral reversed this: the Qwen weaknesses are what training must fix, and he does not like Devstral. Devstral is not used further.
• Devstral Small 2 tested and dropped (same settings, 11 tasks): mean 6.8 vs Qwen 15.8; 1/11 vs 3/11 tasks above 0; end reason loop 6 vs 7 (no lower loop rate), tool_budget 3 vs 3. Results: runs/archive/devstral/. The Devstral weights were deleted. No further base model tests.
• Official Qwen baseline (complete, 11 tasks, thinking off, max_tokens 16384, budget 60, guard 3, MacBook, run base 22000): runs/stage1/baseline.json (= baseline_qwen.json). Comparison: docs/stage1-baseline.md. Older partial Qwen runs (Mac mini, aborted or thinking on): runs/archive/qwen38/, not comparable.
• Training tool is not chosen. Mac: mlx_lm.lora (small test). Rented GPU: open (Unsloth, TRL + PEFT, Axolotl).

7. Next steps

1. Test one H slot (G0168) and one K slot (G0178); then run step F (python3 -m harness.evalset run).
2. Own-test scoring part (a): run the model's own tests against faulty/ mutants.
3. Step F: eval set of 110 tasks: ~150 candidates, empirical filter (2–3 models), Claude review with a checklist, Kral spot-checks ~10 flagged tasks.
4. Scoring for stop tasks (category H) and category K (free-text input, other tool schema).
5. Step 1.3b: generic ABAP MCP interface (spec abap-mcp-arayuz.md, later). The proxy is the first adapter.

8. Open items for Kral (EPOD server)

• When a write fails, run a syntax check and return its messages. Now the server says only "save failed" (for example for TYPE c LENGTH n in a method signature). The model cannot see the cause and cannot repair; this hurts eval and training. Workaround in the generator: local abaplint parser check.
• Unit test timeout: an endless loop in tested code blocks the one RFC connection for ~8 min (mutation run 5201, G0002). Request: stop a unit test run after N seconds (DURATION SHORT = 60 s).
• Concurrency: queue calls per RFC connection, or use a connection pool.
• BDEF creation (needed for RAP tasks).