Making etcd incidents easier to debug in production Kubernetes

Diagnosing and Recovering etcd: Practical tools for Kubernetes Operators Why etcd incidents are so hard to reason about From symptoms to clarity with etcd-diagnosis Quick checks vs. deep diagnostics Understanding common etcd failure modes Recovery is a last resort, and that’s intentional Building calmer, more predictable operations References Posted on March 12, 2026 by Natalie Fisher and Benjamin Wang, Broadcom CNCF projects highlighted in this post When Kubernetes clusters experience serious issues, the symptoms are often vague but the impact is immediate. Control plane requests slow down. API calls begin to time out. In the worst cases, clusters stop responding altogether. More often than not, etcd sits at the center of these incidents. Because etcd is both small and critical, even minor degradation can cascade quickly. And when something goes wrong, operators are frequently left piecing together logs, metrics, and tribal knowledge under pressure. The goal of the recent work around etcd diagnostics and recovery is simple: help platform teams move faster from symptom to signal and only reach for recovery when it’s truly necessary. This post walks through the motivation behind that work, introduces the etcd-diagnosis tooling, and explains how it fits into real-world Kubernetes operations, including environments like vSphere Kubernetes Service (VKS). etcd failures rarely announce themselves clearly. Instead, operators tend to encounter messages like: apply request took too long etcdserver: mvcc: database space exceeded apply request took too long etcdserver: mvcc: database space exceeded These errors don’t immediately tell you why the system is unhealthy.