RepoJournal
Hugging Face

@huggingface

Transformers, Datasets, and the open AI-model layer

Keep up with Hugging Face in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

One email a day. Unsubscribe in one click.

Pick a date

  1. Thu 18 jun
  2. Sat 27 jun
  3. Sun 28 jun
  4. Mon 29 jun
  5. Tue 30 jun
  6. Wed 1 jul
  7. Thu 2 jul
  8. Fri 3 jul
  9. Sat 4 jul
  10. Sun 5 jul
  11. Mon 6 jul
  12. Tue 7 jul
  13. Wed 8 jul
  14. Thu 9 jul
  15. Fri 10 jul
  16. Sat 11 jul
  17. Sun 12 jul
  18. Mon 13 jul
  19. Tue 14 jul
  20. Wed 15 jul
  21. Thu 16 jul
  22. Fri 17 jul
  23. Sat 18 jul
  24. Sun 19 jul
  25. Mon 20 jul
  26. Tue 21 jul
  27. Wed 22 jul
  28. Thu 23 jul
  29. Fri 24 jul
  30. Today 25 jul
Topics: Python AI / ML Full archive →

The Wire · Showcase

SWIFT TOKENIZERS FIXED FOR UNICODE, TRANSFORMERS CI PATCHED, MONGOKU SHIPS SCHEMA AUDIT

By RepoJournal · Filed · About Hugging Face

apocryphx tarekziade coyotte508 HuggingFaceInfra
4 people shipped this
apocryphx @apocryphx 3 cited tarekziade @tarekziade 1 cited coyotte508 @coyotte508 1 cited HuggingFaceInfra @HuggingFaceInfra 1 cited

Swift-transformers shipped three critical tokenizer fixes that resolve grapheme cluster bugs breaking emoji and combining marks across Unigram, BPE, and BasicTokenizer [ref:8] [ref:9], while transformers CI dodged an E2BIG argument limit blowup on large PRs [ref:1].

The Swift fixes address a fundamental mismatch: SentencePiece vocabularies index by Unicode scalar, but Swift's Character type operates on extended grapheme clusters, causing emoji like '1️⃣' and combining marks in Thai, Devanagari, and Japanese to fail tokenization [4] [5] [6]. These aren't edge cases - they're vocab coverage holes that would silently produce wrong tokens in production. Transformers sidestepped a CI disaster by refetching PR files in-script instead of piping them through environment variables, which was hitting the kernel's MAX_ARG_STRLEN limit on PRs with large patches [7]. Mongoku merged a schema auditing feature with a medium-risk warning: new collection-wide introspection endpoints could surface expensive aggregations that timeout on large datasets [8]. Hub-docs auto-bumped inference provider packages and regenerated docs without incident [9].

One email a day. Unsubscribe in one click.

Action items

References

  1. [1] Unigram lattice walks Unicode scalars (#352, Bug 3) (#356) huggingface/swift-transformers
  2. [2] BPE merge by Unicode scalar, not grapheme cluster (#352, Bug 4) (#355) huggingface/swift-transformers
  3. [3] bugfix(ci): avoid E2BIG in pr_slow_ci_suggestion (#45983) huggingface/transformers
  4. [4] Unigram lattice walks Unicode scalars (#352, Bug 3) ↗ huggingface/swift-transformers
  5. [5] BPE merge by Unicode scalar, not grapheme cluster (#352, Bug 4) ↗ huggingface/swift-transformers
  6. [6] Strip Japanese voiced-kana marks in BasicTokenizer (#352, Bug 2) ↗ huggingface/swift-transformers
  7. [7] bugfix(ci): avoid E2BIG in pr_slow_ci_suggestion ↗ huggingface/transformers
  8. [8] Add schema auditing endpoints and navigation tab ↗ huggingface/Mongoku
  9. [9] [Bot] Update Inference Providers documentation ↗ huggingface/hub-docs

Quick answers

What shipped in Hugging Face on May 17, 2026?
Swift-transformers shipped three critical tokenizer fixes that resolve grapheme cluster bugs breaking emoji and combining marks across Unigram, BPE, and BasicTokenizer [ref:8] [ref:9], while transformers CI dodged an E2BIG argument limit blowup on large PRs [ref:1]. In total, 6 commits, 7 pull requests, and 1 releases landed.
Who contributed to Hugging Face on May 17, 2026?
4 developers shipped this update, including apocryphx, tarekziade, coyotte508, and HuggingFaceInfra.
What were the notable Hugging Face updates?
Unigram lattice walks Unicode scalars (#352, Bug 3) (#356), BPE merge by Unicode scalar, not grapheme cluster (#352, Bug 4) (#355), and bugfix(ci): avoid E2BIG in pr_slow_ci_suggestion (#45983).

More from @huggingface

Hugging Face @huggingface

TRANSFORMERS OVERHAULS LINEAR ATTENTION WHILE DEPRECATING LEGACY RESPONSE SCHEMA

The transformers library is retiring its fragile response_schema prototype in favor of streaming-compatible parsing, while simultaneously refactoring every linear attention model to use standardized convolution patterns.

Matt Cyril Vallez Anton Vlasjuk qgallouedec +5
python 70 shipped 2-min read
Hugging Face @huggingface

TRANSFORMERS SHIPS FSDP DISTRIBUTED TRAINING STACK, HUB LIBRARY PLUGS REDOS HOLE

Hugging Face landed distributed training orchestration in transformers while plugging a regex vulnerability that could stall untrusted card parsing for minutes.

evalstate Ferdinand Mom Kashif Rasul Harry Mellor +10
python 61 shipped 1-min read
Hugging Face @huggingface

TRANSFORMERS HARDENS AGAINST PYTORCH FRAGMENTATION WHILE TRL SIMPLIFIES DISTILLATION

Transformers plugged a cascading import failure that breaks downstream CI on older PyTorch versions, while TRL rips out dead code to lock DistillationTrainer into prompt-only datasets.

dacorvo tarekziade vbayanag eustlb +5
python 70 shipped 2-min read
Hugging Face @huggingface

DATASET VIEWER LOCKS DOWN ARROW, FUNES SHIPS GROUNDED ASK

Hugging Face security teams moved overnight to contain a critical Arrow IPC parsing vulnerability in dataset-viewer while shipping three production hardening releases across Repo2RLEnv, optimum-executorch, and funes.

Remy XciD lhoestq Adithya S K +4
python 91 shipped 2-min read

Daily updates, in your inbox

Follow Hugging Face

Keep up with Hugging Face in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.

or

One email a day. Unsubscribe in one click. Read a past issue →

Elsewhere on the wire

AI Agents AI Agents

CLAUDE OPUS 5 LANDS ACROSS THE STACK

The newest Anthropic model is now live in langchain, Cline, and llama-index, with native support for extended reasoning and 1M context windows.

github-actions[bot] hntrl dependabot ak684 +6
ai-agents 28 shipped 1-min read
Local LLMs Local LLMs

OLLAMA LANDS LAGUNA SUPPORT AND CRUSHES MEMORY LEAKS WHILE SGLANG HITS V0.5.16 WITH CONFIDENCE-DRIVEN SPECULATIVE DECODING

Ollama shipped three critical performance and reliability fixes for Metal residency and concurrent access patterns, while SGL-Lang released 0.5.16 with a new speculative algorithm hitting 383.7 tok/s on DeepSeek-V4.

dhiltgen ParthSareen Xuan-Son Nguyen Johannes Gäßler +11
llms 210 shipped 2-min read
CachyOS @CachyOS

HYPRLAND V0.56 FIXES LAND, PACKAGE ECOSYSTEM ROLLS FORWARD

Hyprland configuration updated for v0.56 compatibility across multiple desks, while the AUR-derived ecosystem locked in four automated package bumps.

cantosun99 Jinra Aarrayy Valeri +7
infra 85 shipped 1-min read
Elixir & Phoenix Elixir & Phoenix

LIVEVIEW ASYNC CLEANUP FIX SHIPS ALONGSIDE RANGE OPTIMIZATIONS

Phoenix LiveView closes a critical async task test failure while Elixir cuts unnecessary abs calls from Range operations.

Barna Kovacs Alexander Gubarev SteffenDE José Valim +1
elixir 19 shipped 1-min read
Want every project, not just this one?