$ the-wire · showcase
MILES Hero Small defaults move to 224 requests, plus an a4b SFT recipe and a fix for a tool-data tokenizer crash
By RepoJournal · Filed · About Alignment & Post-training · Composed from the cited sources · methodology
Three things an operator can act on today: the OLMo 3.5 Small MILES examples now ship the best measured serving setting (224 running requests, 3600 s request timeout), a standing crash on tool-call SFT data gets a fix, and a new a4b recipe lands next to c12r.
MILES: Hero Small 4-node example to 224 running requests + 3600 s request timeout (-13% vs 160, -31% vs 96) allenai/open-instruct
Both MILES example configs for Hero Small now carry the best measured serving setting: admission R=224 via sglang_max_running_requests with matching CUDA graph decode batch size, 256 mamba/KDA slots, server concurrency 448 and sglang_mem_fraction_static 0.92, plus [core] refresh_request_timeout = 3600.0. The PR reports a 13% gain over 160 requests and 31% over 96, so if you copied the old four-...
Decode JSON-string tool-call arguments before rendering SFT rows allenai/open-instruct
SFT datasets store each tool call's arguments as a JSON string, and the Olmo 3.5 chat template iterates them with | items, so main's tokenizer raises TypeError: Can only get item pairs from a mapping on tool data. The PR reports 295 of the first 300 rows failing on A4B's Nemotron tool source, and all 300 passing once the strings are decoded first; the other four A4B sources tokenize the same ei...
SFT: add A4B (OLMo 3.5 Small) recipe pinned to the A4B image allenai/open-instruct
scripts/train/sft.py gains an a4b recipe, runnable as uv run python scripts/train/sft.py --recipe a4b, modelled on c12r: same C12 data and cache, 10% long-context replay, half epoch at 18,410 steps on 2x8 B300. What differs from c12r is the base (Olmo 3.5 Small hero at its 4T decay endpoint) and peak LR 7.3e-5.
Add a CUDA 13 MoE image with transformer-engine for MoE v2 SFT allenai/open-instruct
MoE v2 SFT needs transformer-engine for its token-permutation kernels and no main image shipped it, which is why A4B runs have been built from the a4b-bigbatch branch. build_image_and_launch.sh --cuda-version 13 --moe now builds a -cuda13-moe image with transformer-engine[pytorch]==2.16.1 on a CUDA 13.3.1 + cuDNN base. Still a draft.
Action items
- → Update MILES configs/miles/examples/small/four-node.toml to sglang_max_running_requests 224 and refresh_request_timeo... allenai/open-instruct [plan]