The Wire · Showcase
SPEECH-TO-SPEECH PROTOCOL OVERHAUL SHIPS, DIFFUSERS FIXES LORA STATE CORRUPTION
By RepoJournal · Filed · About Hugging Face
The speech-to-speech realtime pipeline just rebuilt its entire response lifecycle to keep audio, transcripts, and tool calls synchronized across WebSocket and WebRTC, while diffusers patched a critical bug where partially unfused LoRA adapters were marked as clean but remained fused in silent components.
The speech-to-speech team landed four interconnected protocol fixes that address the root cause of output lifecycle fragmentation [1][2][3][4]. The changes reserve stable assistant output identities at audio start, align transcript event semantics to emit full completion signals, introduce an explicit backend registry to eliminate conditional duplication, and populate `response.done` with the actual generated output items so clients get a coherent view of what the assistant produced. This is the protocol stabilization that makes tool-calling reliable at scale. On the diffusers side, Aloys Jehwin fixed a state corruption bug where `unfuse_lora(components=[...])` with a component subset would drop adapters from `_merged_adapters` as soon as they unfused from the first component, even though they remained physically baked into the others [5]. The fix recomputes `_merged_adapters` by scanning all loadable components after unmerge. Sayak Paul also fixed torchao test failures [6]. The kernels team opened kernel-kernel dependency support: danieldk added `kernel-depends` to `build.toml` metadata across kernels-data and kernels repos, laying groundwork for kernel composition [7][8], and started testing einops as the first kernel with inter-kernel dependencies [9]. TRL squashed a GRPO loss aggregation bug where the `luspo` branch never applied per-token completion masks elementwise [10] and excluded a buggy TruffleHog detector that was triggering false positives on ordinary 35-character test names [11].
One email a day. Unsubscribe in one click.
Keep up with Hugging Face in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
One email a day. Unsubscribe in one click. Read a past issue →
Action items
- → Review speech-to-speech protocol changes if you consume realtime streams - transcript and tool-call event shapes changed [ref:6] huggingface/speech-to-speech [plan]
- → Verify LoRA state after partial unfuse in production pipelines - the corruption may be silent huggingface/diffusers [monitor]
- → Update GRPO training runs if you use luspo loss with beta != 0.0 or multi-token sequences huggingface/trl [plan]
References
- [1] Harden Realtime response output lifecycle ↗ huggingface/speech-to-speech
- [2] Align Realtime transcript event semantics ↗ huggingface/speech-to-speech
- [3] Introduce backend registry for pipeline construction ↗ huggingface/speech-to-speech
- [4] Populate response.done with generated output items ↗ huggingface/speech-to-speech
- [5] fix(lora): only drop adapter from _merged_adapters when unfused from all components (#14385) huggingface/diffusers
- [6] [tests] fix torchao tests (#14258) huggingface/diffusers
- [7] Add `kernel-depends` option to `general` options in `build.toml` ↗ huggingface/kernels
- [8] Add `kernel-depends` option to `general` options in `build.toml` (#749) huggingface/kernels
- [9] einops: add ↗ huggingface/kernels-community
- [10] [GRPO] Apply the completion mask elementwise in the luspo loss aggregation ↗ huggingface/trl
- [11] ci: exclude the buggy TruffleHog lob detector ↗ huggingface/trl