The Wire · Showcase
TRANSFORMERS SPLITS VLLM CI ACROSS DEDICATED RUNNERS; SENTENCE-TRANSFORMERS FIXES FLOAT PRECISION ACROSS ALL RANKING
By RepoJournal · Filed · About Hugging Face
Transformers restructured its vLLM test pipeline to eliminate OOM failures while sentence-transformers shipped three critical precision fixes in float16/bfloat16 compute paths.
The transformers team moved vLLM CI to dedicated runners [1] [2], splitting test_initialization into an 8-shard matrix job to fix persistent out-of-memory errors that plagued sequential runs. This is the isolation win the team needed to keep vLLM test coverage in the pipeline without blowing up resource usage. On the sentence-transformers side, three separate fixes hit v6 simultaneously: CrossEncoder.rank now returns native Python floats instead of numpy.float32 scalars [3], which makes results JSON-serializable and stops rank comparisons from breaking on device tensors. More critically, both similarity computation [4] and cross-encoder logits [5] now upcast to float32 before activation, preventing reduced-mantissa formats from coarsely bucketing scores when models run in float16 or bfloat16. The team also exposed device and chunking parameters in MaxSim similarity functions to speed CUDA computation by up to 39x on large embeddings [6]. In diffusers, LTX25AutoBlocks got refactored into its own file with a dedicated model name [12] [14], and keep_in_fp32 module tests were tightened [13]. Speech-to-speech fixed Whisper STT handlers that were incorrectly finalizing turns on progressive audio [9] and rewrote language detection to handle recent transformers releases that no longer echo decoder prefixes [10]. Muse Glimmer is now exportable [7] [8], and the realtime protocol now overlaps tool-result generation [11].
One email a day. Unsubscribe in one click.
Keep up with Hugging Face in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
One email a day. Unsubscribe in one click. Read a past issue →
Action items
- → Upgrade sentence-transformers immediately if using float16/bfloat16 models with MaxSim or CrossEncoder ranking huggingface/sentence-transformers [immediate]
- → Review vLLM test sharding strategy and monitor CI runtime reduction on your transformers fork huggingface/transformers [plan]
- → Test Whisper STT progressive audio transcription if running speech-to-speech in realtime mode huggingface/speech-to-speech [plan]
References
- [1] CI: add vllm-test-init and vllm-test-transformers jobs on dedicated runners ↗ huggingface/transformers
- [2] CI: add vllm-test-init and vllm-test-transformers jobs on dedicated runners (#47934) huggingface/transformers
- [3] [`v6`] refactor: return Python floats from CrossEncoder.rank ↗ huggingface/sentence-transformers
- [4] [`v6`] fix: compute similarity scores in float32 to avoid low-precision ties (HPS) ↗ huggingface/sentence-transformers
- [5] [`v6`] fix: upcast cross-encoder logits to float32 before activation (HPS) (#3893) huggingface/sentence-transformers
- [6] [`v6`] Forward similarity kwargs to similarity functions, expose MaxSim device & chunking ↗ huggingface/sentence-transformers
- [7] Make muse glimmer exportable ↗ huggingface/transformers
- [8] Make muse glimmer exportable (#47871) huggingface/transformers
- [9] fix(stt): stop Whisper handlers finalizing turns on progressive audio ↗ huggingface/speech-to-speech
- [10] Fix Whisper STT language detection reading a text token ↗ huggingface/speech-to-speech
- [11] Overlap tool follow-up generation within the Realtime protocol ↗ huggingface/speech-to-speech
- [12] Move `LTX25AutoBlocks` to new file with new model name ↗ huggingface/diffusers
- [13] [tests] tighten keep_in_32 modules tests (#14399) huggingface/diffusers
- [14] Move `LTX25AutoBlocks` to new file with new model name (#14453) huggingface/diffusers