$ the-wire · showcase
unicode/norm composition bugs shake the foundation
By RepoJournal · Filed · About Go
12 people shipped this
Russ Cox
@rsc
4 cited
Robert Griesemer
@griesemer
2 cited
Damien Neil
@neild
2 cited
Peter Weinberger
@pjweinbgo
1 cited
Timo Furrer
@timofurrer
Alan Donovan
@adonovan
Alex Putman
@aputman
Ruslan Semagin
@pixel365
Tianon Gravi
@tianon
Tobias Klauser
@tklauser
Lorenz Brun
@lorenz
Nicholas S. Husin
@nicholashusin
Russ Cox landed a five-part fix for Unicode normalization that silently corrupted text, and gopls is hardening inlining, but the real headline is the type-checking cleanup rolling through go/types.
Russ Cox's unicode/norm fixes [1] are the story this morning: the reorderBuffer was silently dropping table-driven compositions whenever a segment started with a Hangul syllable [1], and a truncated recomposition map key caused supplementary-plane runes to alias BMP entries, breaking canonical equivalence [2]. Damien Neil cleaned up two more edge cases [3] [4] to round out the series. On the net desk, Russ also patched two HTTP/3 landmines: a nil TLSConfig in newQUICConfig that panicked on default transports [5], and a leak where RoundTrip could return before the request body was fully closed [6]. "The result is a composition that is not canonically equivalent to its input, reachable from plain" [2] — and yes, that's as bad as it sounds. Gopls now rejects inlining for variables that are reassigned or have their address taken [7], closing golang/go#77543, while go/types continues its steady march toward cleaner type inference with the target refactor [8] [9].
Action items
- → Pull the unicode/norm fixes into any project using golang.org/x/text for domain names, filenames, or user input normalization golang/text [immediate]
- → Update golang.org/x/net to pick up the HTTP/3 nil TLSConfig fix if you use the default http.Transport with http3 golang/net [immediate]
- → Watch gopls inlining changes — they now refuse to inline modified variables, which could change refactoring behavior golang/tools [plan]
- → Monitor the go/types target refactor — it's a large change, but if you rely on type inference for composite literals, test against the tip golang/go [monitor]
References
- [1] unicode/norm: compose non-Hangul runes after a Hangul syllable golang/text
- [2] unicode/norm: don't truncate runes in the recomposition map key golang/text
- [3] unicode/norm: simplify short source detection in Form.transform golang/text
- [4] unicode/norm: prevent decomposeSegment from moving backwards golang/text
- [5] internal/http3: handle a nil TLSConfig in newQUICConfig golang/net
- [6] internal/http3: close the request body before RoundTrip returns golang/net
- [7] gopls/internal/golang: reject inlining for modified variables golang/tools
- [8] go/types, types2: allow target to hold any type, not just signatures golang/go
- [9] go/types, types2: remove all vestiges of composite literal hint parameters golang/go
