$ the-wire · showcase
CONSUL SERVERS LOSE THEIR MINDS AFTER RENAME
By RepoJournal · Filed · About HashiCorp · Composed from the cited sources · methodology
![dependabot[bot]](https://avatars.githubusercontent.com/in/29110?v=4)
A critical bug in Consul server rejoining after maintenance leaves clusters unable to find their Raft leader, breaking peer lookups and causing silent failures across production.
Consul shipped a fix [1] for a lurking incident where renaming and rejoining a server creates a phantom IP collision that breaks the entire Raft consensus layer. You'll see the error "Raft leader not found in server lookup mapping" followed by cascading 500s when the cluster can't locate its peers. This isn't theoretical, it's a known production issue that the team debugged after it happened to them. On the observability front, consul-esm rolled out test coverage reporting [2] across its CI pipeline, borrowing the working approach from consul-dataplane to catch regressions earlier. The team also landed a workaround for stale read node configuration [3] that was blocking their badge workflow, fixing a CI footgun where running coverage twice on the same branch would fail. Finally, action-setup-bob bumped its checkout action [4] to 6.0.3, a routine maintenance update across the HashiCorp GitHub Actions ecosystem.
Action items
- → Review and test the server IP collision fix in your staging environment before deploying to production hashicorp/consul [immediate]
- → Audit any recent server renames or maintenance operations for the symptoms described in the fix hashicorp/consul [plan]
- → Monitor Raft consensus health if running Consul at scale hashicorp/consul [monitor]
References
- [1] Fix server ip collision ↗ hashicorp/consul
- [2] ci: add Go test coverage reporting ↗ hashicorp/consul-esm
- [3] ci: decouple stale_read_nodes config fix into separate PR (#380) ↗ hashicorp/consul-esm
- [4] Bump actions/checkout from 6.0.2 to 6.0.3 ↗ hashicorp/action-setup-bob
