The Wire · Showcase
KOPS DROPS KINDNET GRID JOBS AMID DNS FAILURES
By RepoJournal · Filed · About Kubernetes
The kops test grid is shedding 27 kindnet periodics after a node-local DNS cache started failing live Services, and conformance timeouts are getting a much-needed bump.
kops' kindnet grid jobs for version 1.34 are being dropped entirely [1], a move that eliminates 27 periodics (12 AWS, 15 GCE) after the node-local DNS cache intermittently answered NXDOMAIN for live Services, failing a [sig-network] spec with "service is not reachable within 2m0s timeout" [2]. The cache, enabled by kops' own manifest flags, has been defaulted off upstream since kindnet 629dc20ec6 [1]. Meanwhile, the kops conformance suite no longer fits in its 150m timeout, causing the Go test binary to panic and produce no junit at all, so the team is raising the timeout to 200m and quieting test nodes [3]. On the tooling front, a new dump capture for Calico BGP session state aims to leave evidence behind for the GCE Calico grid failures, addressing the dead end where artifacts only showed symptoms [4]. Also, a live race in nodeup is being fixed: systemd-networkd restarts can land mid-apt-get, and the new ordering prevents that [5]. In other news, the v1.37 Release Announcement Blog is in progress [6], and KEP-5207 has been marked as implemented [7].
One email a day. Unsubscribe in one click.
Keep up with Kubernetes in about 3 minutes a day: what actually shipped — the commits, pull requests, releases, and security advisories that matter.
One email a day. Unsubscribe in one click. Read a past issue →
Action items
- → Review kops 1.34 grid changes if you rely on kindnet periodics kubernetes/test-infra [monitor]
- → Upgrade kops conformance jobs to 200m timeout to avoid blank testgrid kubernetes/test-infra [plan]
- → Test the Calico BGP dump capture in kops toolbox dump kubernetes/kops [plan]
References
- [1] kops: drop the kindnet grid jobs for kops 1.34 kubernetes/test-infra
- [2] kops: drop the kindnet grid jobs for kops 1.34 ↗ kubernetes/test-infra
- [3] kops: raise the conformance timeout to 200m and quiet the test nodes ↗ kubernetes/test-infra
- [4] dump: capture Calico BGP session state from each node ↗ kubernetes/kops
- [5] nodeup: order systemd-networkd restarts after package installs ↗ kubernetes/kops
- [6] Kubernetes v1.37 Release Announcement Blog ↗ kubernetes/website
- [7] KEP-5207: marked as implemented ↗ kubernetes/enhancements