The Wire · Showcase
NODE-PROBLEM-DETECTOR FIXES CRITICAL KMSG WATCHER DEADLOCK
By RepoJournal · Filed · About Kubernetes
A cascading restart bug in node-problem-detector's kmsg watcher caused 37k restarts in 200ms and could deadlock the shutdown path, and the fix just landed.
The kmsg watcher restart mechanism had three interlocking failure modes that only showed up under load [1]. The worst: a parser whose channel closes right after a successful restart triggered a hot restart loop with zero delay, measured at 37k restarts in 200 milliseconds. A second issue deadlocked Stop() when logCh filled up and the consumer was already shutting down, blocking watchLoop on a plain send. The third was a double-close on deferred cleanup when Stop() interrupted a restart attempt. All three are fixed in the same PR [2]. In parallel, node-problem-detector shipped three releases: v1.34.4, v1.35.3, and v1.36.0 [3], with v1.36.0 including the kubelet service name pattern update and dependency bumps [4]. The infrastructure team is also pivoting IBM ppc64le boskos resources from the deprecated wdc06 datacenter to the eu-de-1 region [5], which will improve test reliability for power-based architectures. Minikube landed a cleaner dependency consolidation, replacing golang.org/x/mod/semver with the already-present blang/semver library for Helm validation [6], plus routine addon updates across inspektor-gadget, kong, and cloud-spanner.
Action items
- → Update node-problem-detector to v1.36.0 before next cluster upgrade kubernetes/node-problem-detector [plan]
- → Review kmsg watcher logs for restart storms on existing deployments kubernetes/node-problem-detector [monitor]
- → Bump minikube dependency consolidation into next release kubernetes/minikube [plan]
References
- [1] Fix kmsg watcher restart and shutdown issues ↗ kubernetes/node-problem-detector
- [2] Merge pull request #1311 from hakman/kmsg-watcher-fixes kubernetes/node-problem-detector
- [3] Promote node-problem-detector v1.34.4, v1.35.3, v1.36.0 ↗ kubernetes/k8s.io
- [4] v1.36.0 ↗ kubernetes/node-problem-detector
- [5] Replace wdc06 DC with eu-de-1 region in boskos on ibm ppc64le ↗ kubernetes/k8s.io
- [6] addons: use blang/semver for helm version validation ↗ kubernetes/minikube