$ the-wire ยท showcase
Tensor-parallel embeddings get their own styles, and LoRA silently skips MoE expert layers
By RepoJournal ยท Filed ยท About Hugging Face ยท Composed from the cited sources ยท methodology
13 people shipped this
Ilyas Moutawwakil
@IlyasMoutawwakil
1 cited
jiqing-feng
@jiqing-feng
1 cited
albertvillanova
@albertvillanova
1 cited
sywangyi
@sywangyi
1 cited
SunMarc
@SunMarc
1 cited
Tarek Ziade
@tarekziade
Ting SUN
@Sunt-ing
Steven Liu
@stevhliu
Younes Belkada
@younesbelkada
AmineDiro
@AmineDiro
Sergio Paniego Blanco
@sergiopaniego
danieldk
@danieldk
hf-kernels-bot[bot]
@hf-kernels-bot[bot]
Transformers split embedding sharding out of the linear-layer styles and made longrope compileable, while TRL started warning about a LoRA configuration that quietly trains only routed experts on MoE models under transformers v5 with peft 0.19 to 0.21.2.
๐จ TP: give embeddings their own style, shard input embeddings by default, and shard widths that don't divide by the TP size (#49570) huggingface/transformers
ColwiseParallel and RowwiseParallel no longer special-case nn.Embedding and now require a linear layer; embeddings move to EmbeddingRowwiseParallel (embedding_rowwise, vocab-parallel) and EmbeddingColwiseParallel (embedding_colwise, hidden-dim parallel, used by qwen4_exp). If your tensor-parallel plan shards nn.Embedding through the old colwise/rowwise styles, it needs rewriting before it runs ...
Warn when LoRA skips the dense and shared expert layers of MoE models on transformers v5 with peft 0.19 to 0.21.2 huggingface/trl
The trainers now warn when a LoRA config would skip the dense and shared expert layers of a MoE model. The cause: with transformers v5 and peft 0.19 to 0.21.2, target_modules like ["gate_proj", "up_proj", "down_proj"] or "all-linear" train only the routed experts of models with a dense first layer or shared experts (GLM-4-MoE, DeepSeek-V3, and others), so those layers silently stay frozen.
Make longrope compileable (#49555) huggingface/transformers
longrope_frequency_update picks between short and long scaling factors from seq_len, which is derived from position_ids and is therefore always a tensor. That data-dependent branch failed fullgraph compilation on the first forward pass; the fix computes both sets, so longrope checkpoints now compile.
Fix XPU varlen LSE indexing for padded queries huggingface/kernels-community
XPU varlen attention can receive query tensors whose allocated length exceeds the real token count (64 slots, 5 valid tokens), and the LSE handling could index past the valid range. Forward LSE writes and backward LSE reads now use [num_heads, total_q] token layout while the backward softmax_d offset keeps its batch layout, with a padded-query regression test.
metal flash sdpa flash attn func huggingface/kernels-community
Also landing: metal-flash-sdpa gained flash_attn_func, without which any unpadded input (a single prompt, a plain generate) failed in transformers with TypeError: 'NoneType' object is not callable; TRL's AsyncGRPO can now cap in-flight rollouts on the vLLM server's actual KV cache usage instead of the fixed max_inflight_tasks; and tokenizer tests moved to hf-internal-testing/tiny-random-mluke, ...
Action items
- โ Audit LoRA target_modules on MoE models against TRL's new warning before your next training run huggingface/trl [plan]
- โ Migrate tensor-parallel plans that shard nn.Embedding via ColwiseParallel or RowwiseParallel to embedding_colwise or ... huggingface/transformers [plan]
