The Wire · Showcase
TRANSFORMERS-CI SHIPS INTELLIGENT TEST TRIAGE, CUTS LLM NOISE 100X
By RepoJournal · Filed · About Hugging Face
Hugging Face's serge verify loop now distills 100KB failure tracebacks down to the actual assertion, runs every test 5 times to catch flakiness, and dispatches each failure category with a custom instruction before spinning GPU or consuming a single LLM token.
The CI team landed three precision moves across transformers-ci [1] [2] [3]. First, serge verify now caps tensor dumps in pytest longrepr output, keeping the asserted slice (a handful of numbers) and dropping tens of kilobytes of intermediate model-output noise like logits and vision_model_output [4]. Second, triage workflows adapt their instruction, reproduce strategy, and trace budget per failure category (OOM, output_mismatch, cuda_runtime, load_error, import_or_config, other) before any GPU minute or LLM token is spent [5]. Third, the verify loop runs each targeted test 5 times per tree with a fresh pytest process each run, requiring the red-to-green result to hold on every run to rule out flakiness [3]. In parallel, transformers landed a tied-embedding fix for tensor parallelism inference [6] [7], and tau shipped multimodal image serialization for vision models reading files from disk [8] [9]. Diffusers completed a test refactor sweep across all four CogVideoX pipeline variants (text-to-video, image-to-video, video-to-video, fun control) migrating to unified mixin structure and moving integration tests to pytest slow class [10] [11] [12] [13].
One email a day. Unsubscribe in one click.
Keep up with Hugging Face in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
One email a day. Unsubscribe in one click. Read a past issue →
Action items
- → Review transformers-ci failure triage output in next CI run; verify distilled tracebacks reach the LLM without truncation huggingface/transformers-ci [plan]
- → Merge hf-mcp-server 0.4.0 release once refactors (legacy tools, stateful HTTP, audio transform) are validated in staging huggingface/hf-mcp-server [plan]
- → Monitor CogVideoX test migrations for memory regression; confirm cache mixin coverage matches old test suite huggingface/diffusers [monitor]
References
- [1] serge verify: distill failure tracebacks (cap tensor dumps, keep the assert) ↗ huggingface/transformers-ci
- [2] triage: adapt the agent workflow to each failure category ↗ huggingface/transformers-ci
- [3] serge verify: run targeted tests N times to rule out flakiness ↗ huggingface/transformers-ci
- [4] serge verify: distill failure tracebacks (cap tensor dumps, keep the assert) (#42) huggingface/transformers-ci
- [5] triage: adapt the agent workflow to each failure category (#40) huggingface/transformers-ci
- [6] Fix TP inference for tied embedding (#47503) huggingface/transformers
- [7] Fix TP inference for tied embedding ↗ huggingface/transformers
- [8] Send read-tool images to vision models ↗ huggingface/tau
- [9] Send read-tool images to vision models (#479) huggingface/tau
- [10] refactor cogvideox pipeline tests to the new mixin structure ↗ huggingface/diffusers
- [11] refactor cogvideox fun control pipeline tests to the new mixin structure ↗ huggingface/diffusers
- [12] refactor cogvideox image to video pipeline tests to the new mixin structure ↗ huggingface/diffusers
- [13] refactor cogvideox video to video pipeline tests to the new mixin structure ↗ huggingface/diffusers