Pain point · Reliability & bugs · Causes churn
Stale internode gRPC connections to terminated pods on Kubernetes
3 source threads · first seen 2021-01 · last seen 2026-04
Summary
After scale-down, rolling updates or node drains the servers keep dialing IPs of deleted pods, producing continuous timeouts. The team considers StatefulSets, which would diverge from the official Helm chart.
- Affects
- Teams running the official Helm chart on Kubernetes
- Workaround
- Considering StatefulSets instead of Deployments
Evidence
Excerpts are copied word for word from the source; follow the link to read it in full.
“After upgrading to v1.30.4, the cluster exhibits permanent ringpop membership”
“Temporal servers keep stale internode gRPC connections after Kubernetes pods are terminated (during scale-down, rolling updates, or node drains)”
“it appeared that a frontend service from Temporal cluster A was talking to a matching service from Temporal cluster B”