erhan/abap-llm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

serve.sh back to Qwen 3.8 27B 4-bit

Browse Source 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014aUaQeLnwbb1zTpN7kHeat
This commit is contained in:
Kral
2026-10-04 17:02:59 +02:00
parent 9f85a71ca5
commit a2ba9e7b44
1 changed files with 4 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
train/serve.sh
View File

@@ -1,12 +1,14 @@
#!/bin/sh
# Base model: Devstral Small 2 (no thinking mode, no chat-template args). Serve the base model (no adapter) with mlx_lm.server. Settings: train/README.md.
# Serve the base model (no adapter) with mlx_lm.server. Settings: train/README.md.
# Usage: train/serve.sh [--adapter-path PATH]
# Base model: Qwen 3.8 27B (decision 2026-10-04). Thinking is switched off per request by train/baseline.py.
# Prompt cache limit: without it the cache grew to 7.9 GB in 15 min (2026-10-03); A4H needs 22 GB.
cd "$(dirname "$0")/.."
exec train/.venv/bin/mlx_lm.server \
--model "$HOME/models/Devstral-Small-2-24B-4bit" \
--model "$HOME/models/Qwen3.8-27B-4bit" \
--host 127.0.0.1 --port 8080 \
--temp 0.2 --top-p 0.95 --top-k 20 --min-p 0 \
--max-tokens 32768 \
--prompt-cache-size 4 --prompt-cache-bytes 6000000000 \
--chat-template-args '{"enable_thinking": true, "reasoning_effort": "medium"}' \
"$@"