$ the-wire · showcase
Transformers 5.22.0 ships Mistral Large 4, TRL 1.17.0 fine-tunes it on one node
By RepoJournal · Filed · About Hugging Face · Composed from the cited sources · methodology
Mistral's 1T-parameter MoE model lands in Transformers and TRL within the same release cycle, while both projects clean up the FSDP2 and checkpointing paths underneath it.
v5.22.0: Mistral Large 4 (Le Chonk) huggingface/transformers
Transformers v5.22.0 adds Mistral Large 4, a 1T-parameter natively multimodal MoE with 49B active parameters per token (52B including embeddings and output), trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs. The release is flagged breaking, so check your pin before pulling it into a training stack.
v1.17.0 huggingface/trl
TRL v1.17.0 adds support for mistralai/Mistral-Large-4-1T-A52B with a LoRA SFT example that fine-tunes it on a single node. The accompanying template sync commit exists because TRL matches models by exact chat-template equality: Mistral changed the template before release (default reasoning_effort is now high, and + became ~), which broke assistant_only_loss against the released checkpoint.
GRPO: renormalize log-probs over the top-k/top-p candidate set huggingface/trl
GRPOTrainer now renormalizes log-probs over the candidate set each token was sampled from when vLLM samples with top_k, top_p or min_p, so the importance sampling ratio is no longer biased toward 0. This covers colocate mode only; server mode needs a separate PR because vLLM returns candidate sets solely from /inference/v1/generate, and only with --return-sampling-mask.
Save AsyncGRPO and AsyncDistillation checkpoints in the mixed-precision dtype huggingface/trl
AsyncGRPOTrainer and AsyncDistillationTrainer wrote fp32 checkpoints even with the default bf16=True under FSDP, because accelerate upcasts parameters before sharding and Trainer._save wrote the gathered state dict as-is; loading with dtype="bfloat16" did not help. _save now casts the floating tensors to the mixed-precision dtype.
Shard the parameters kept in float32 in their own FSDP2 group (#49501) huggingface/transformers
Parameters kept in float32 now get their own FSDP2 shard group, with a bf16 training step added to the FSDP2 tests. A related commit makes the tp/ep/fsdp plans build and expand once instead of per call.
feat(recall): rerank any pool larger than the hits huggingface/funes
Funes reranks any recall pool larger than the hit count, replacing a rule that only reranked above four times k, so the defaults (8 of 30) returned the fused order. On funes-memory's 244 questions the answer now lands in the same 8 hits for 128 of them instead of 95. In doc-builder, a new --recent-blog-days flag on populate-search-engine trims the daily Meilisearch sync from roughly 255 requests.
Action items
- → Pin transformers to your current version or verify compatibility before moving to 5.22.0, which is flagged breaking huggingface/transformers [plan]
- → Rerun assistant_only_loss SFT against the released Mistral-Large-4-1T-A52B checkpoint; older template matching fails huggingface/trl [plan]
