$ the-wire · showcase
HF Mount hardens against Hub failures, funes detects secrets per block
By RepoJournal · Filed · About Hugging Face
10 people shipped this
Real-hardware GPU smoke tests return to ML Intern, hf-mount survives Hub rate limits, and funes now pins secrets to the exact block that contains them.
The ML Intern dogfooding session that cost $2.40 of a $3 budget with zero logged metrics exposed gaps in the smoke-test and job-loop design: the sandbox-first CPU test cannot surface an OOM or meaningful steps-per-second, and the parent kept the job loop when it should not. Running on real hardware is being restored, and the job loop is being removed from the parent process; the fixes are still being tested against a real job before they land. [1]
hf-mount hardened its behavior against three transient failure modes reproduced during cross-cloud checkpoint workloads: EIO on stat() while the Hub rate-limits, permanent ENOTCONN when the FUSE daemon dies mid-write, and committed marker files staying invisible to readers. Separately, negative lookup caching is now configurable: a consumer that opens a path before the producer's upload lands used to hit a hardcoded 30-second negative cache, adding 30 s to a hand-off that takes ~2.5 s to upload; the new --negative-ttl-ms flag defaults to 1 second. [2] [3]
In funes, secret scanning changed so each block (text, tool call, tool result) is staged as its own file instead of one concatenated blob. Previously, the line number trufflehog reported was the only way to associate a secret with a block; because trufflehog counts lines in its own decoded output, a key hidden in base64 could shift the line past the end of the blob, stopping indexing, or point at a different block entirely. "Nothing has read it since texts are told apart by file," and the line field is being dropped because the number is counted in trufflehog's decoded output, not funes' stored text. [4] [5] [6]
LeRobot exposes tokenizer configuration for PI0 and PI0.5 models: the new text_tokenizer_name config on PI0Config and PI05Config lets checkpoints use the vocabulary and token-ID mapping from training, with google/paligemma-3b-pt-224 retained as default for compatibility. Also, fixed pi0.5 LoRA fine-tuning: PI05Pytorch names time projections time_mlp_in and time_mlp_out, but default PEFT targets matched the pi0-style action_time_mlp_* pattern, so the LoRA adapter never targeted these layers during pi0.5 fine-tuning. [7] [8]
Action items
- → Update hf-mount to the hardened release when it ships; the fixes cover EIO, ENOTCONN, and negative cache stalls in cross-cloud checkpoint hand-offs. huggingface/hf-mount [plan]
- → Set --negative-ttl-ms when using shared RW bucket mounts for producer/consumer hand-offs to avoid 30-second negative cache delays. huggingface/hf-mount [plan]
- → Check pi0.5 fine-tuning configs are actually targeting time_mlp layers, since the previous default PEFT targets missed them. huggingface/lerobot [monitor]
References
- [1] ML Intern: smoke on real hardware, and keep the job loop out of the parent ↗ huggingface/chat-ui
- [2] fix: resilience to Hub rate limiting, stalled CAS uploads, and unclean shutdown ↗ huggingface/hf-mount
- [3] fix: make the negative-lookup TTL configurable, default 1 s ↗ huggingface/hf-mount
- [4] refactor(scan): drop the line a secret was reported on ↗ huggingface/funes
- [5] fix: scan blocks as individual files ↗ huggingface/funes
- [6] fix(scan): tell texts apart by the file a secret was found in ↗ huggingface/funes
- [7] Make PI0 and PI0.5 text tokenizers configurable ↗ huggingface/lerobot
- [8] fix(pi05): match time_mlp modules in default PEFT targets (#4494) ↗ huggingface/lerobot
