$ the-wire · showcase
Transformers 5.20 ships Muse Spark Open, TRL adds AWS Trainium
By RepoJournal · Filed · About Hugging Face · Composed from the cited sources · methodology
12 people shipped this
qgallouedec
@qgallouedec
2 cited
vasqu
@vasqu
1 cited
Albert Villanova del Moral
@albertvillanova
1 cited
David Corvoysier
@dacorvo
1 cited
brysonjones
@brysonjones
1 cited
guarin
@guarin
Ita Zaporozhets
@itazap
Yih-Dar
@ydshieh
eustlb
@eustlb
Sergio Paniego Blanco
@sergiopaniego
molbap
@molbap
pkooij
@pkooij
Transformers v5.20.0 lands a 816B-parameter multimodal MoE and a diarizing ASR model, while TRL v1.16.0 makes Neuron a supported training backend.
Release v5.20.0 huggingface/transformers
Muse Spark Open arrives as a multimodal mixture-of-experts model taking text, images, audio, and video as input with 816B total parameters, 42B active per token, and a context window of up to 1,048,576 tokens shared across modalities. The release notes describe it as trained for agentic coding work, and the same release adds MOSS-Transcribe-Diarize.
v1.16.0 huggingface/trl
The release notes credit the v1.15 fused LM head with making Neuron tractable; this version closes the remaining gaps and ships a setup guide. One caveat worth knowing before you port a job: Neuron recompiles once per input shape, so the notes say an unpadded run "stalls about a minute on every new sequence".
Add Muse Spark Open: tiny model, tests and 8-node SFT example huggingface/trl
TRL wires up the new model with a tiny-model script, coverage in the VLM tests for SFT, DPO, KTO, chat templates, and the fused LM head, plus a LoRA SFT example of meta-models/Muse-Spark-Open-816B-A42B on 8 nodes of 8 H100s at 14.2 s/step and 40.2 GB per GPU. The tests are version-gated: they skip below transformers 5.20.0.
perf(multi_task_dit): compile the vision encoder and DiT, dedup task strings in the processor huggingface/lerobot
The multi_task_dit training step is dominated by the CLIP vision encoder forward and backward, with roughly 40% of eager GPU time in unfused kernels. This adds the opt-in compile_model and compile_mode knobs the other policies expose and dedups task strings in the processor so the tokenizer and text encoder run once per distinct task instead of once per data row.
fix(add): stage a refresh until its setup is about to run huggingface/funes
A refreshing funes add or funes update used to copy the incoming integration straight into ~/.funes/agents before the trust prompt; declining left the rejected setup and hook scripts live where the agent runs them, under a manifest naming the old release. Provisioning now stages files in a temporary directory and installs them only right before setup runs.
Prevent VLM policies from generating the image placeholder token in GRPO, RLOO and Distillation huggingface/trl
Elsewhere: TRL blocks VLM policies in GRPOTrainer, RLOOTrainer, and DistillationTrainer from sampling their image placeholder token, which previously produced one more image token than image features and crashed training, and fixes activation offloading on torch below 2.4. Also quiet: the CARLA examples drop the client-side observe loop, and leorobot compiles the multi_task_dit vision tower and...
