By the time I could see the cluster, it looked healthy. The service path had no confirmed problem, kube-apiserver answered readiness checks, and the control-plane processes that had restarted so aggressively were now either stable or failing only intermittently.
That was not what the system had looked like during the incident. In the earlier window, kube-controller-manager and kube-scheduler had restarted in bursts. But this was an air-gapped on-premises environment. Direct access was constrained, and the cluster had already moved on by the time an investigation window became available.
The task was no longer to rescue a service that was visibly down. It was to reconstruct a restart storm from the evidence that remained, decide which explanation best fit it, and turn the remaining uncertainty into questions the infrastructure team could actually verify.
This account stops at that handoff. The infrastructure investigation and follow-up are still in progress. What I can describe is how delayed access changed the work, which evidence changed my hypothesis, and why a report with a clear evidence boundary was more honest than a premature root cause.
01 / WE ARRIVED LATE
The system had changed before we could inspect it.
| Phase | What was visible | What it meant |
|---|---|---|
| Incident window | Control-plane processes restarted in bursts; no service-path failure was confirmed. | The control plane was unstable, but the investigation did not begin with live access. |
| Access window | Restarts had become intermittent or stopped; current readiness and processes looked healthy. | The present state did not explain the earlier process lifecycle. |
| Reconstruction | Previous-process logs, termination state, restart history, and infrastructure metrics were aligned. | The failure path could be narrowed without claiming the initiating cause. |
| Handoff | A report separating facts, inference, and open questions moved through the project manager to the infrastructure owner. | The exact lower-layer cause and remediation remained open. |
This is a different operating problem from an incident where the responder controls the infrastructure and watches the failure live. Access latency becomes evidence latency. Events age out, containers are replaced, counters accumulate, and a healthy process starts writing a new story over the old one.
I therefore separated two questions that looked similar on the surface. Was the cluster serving normally during the access window? The evidence said yes. Why had the earlier control-plane processes exited so often? Current health could not answer that question.
02 / RECONSTRUCT THE WINDOW
Current health was not historical evidence.
The current processes answered questions about now. The previous processes were the witnesses to the incident. Their final messages showed kube-controller-manager and kube-scheduler failing to renew leader-election leases and exiting. That explained the restart mechanism, but not the delay that made the renewal miss its deadline.
- Current readiness and current logs established the state during the access window.
- Previous-container logs and termination state preserved the last part of each failed process lifecycle.
- Restart history showed that the earlier burst and the later intermittent state were not the same operating condition.
- A narrow time window made it possible to compare Kubernetes logs with VM and storage metrics without mixing unrelated startup noise.
Following the dependency chain backward gave the investigation its shape. A leader lease is renewed through kube-apiserver. kube-apiserver depends on etcd. etcd, in turn, depends on its members being scheduled and completing network and durable-I/O work in time. Each symptom told me which dependency had failed to answer, not yet why.
03 / CHANGE THE HYPOTHESIS
The logs looked like a disconnect until they looked like a freeze.
My first leading hypothesis was a network interruption. etcd reported delayed heartbeats and leadership disruption, while kube-apiserver requests missed deadlines. Read literally, those messages looked like members had briefly lost contact with one another.
The shape of the delay changed my mind. There was no clean disconnect-and-reconnect boundary. Read coordination, request handling, and heartbeats appeared to stall together. That looked less like one broken connection and more like a process or virtual machine that had stopped making timely progress.
I did not rule out the network, and I did not prove that storage caused the restart. I changed the leading hypothesis because the freeze-like behavior and cross-role latency explained more of the evidence at once. Host scheduling, a guest pause, a virtual block queue, a backing storage service, or another shared path could still produce the same view from inside the VMs.
04 / STOP AT THE EVIDENCE BOUNDARY
The propagation path was stronger than the root-cause claim.
I used the timestamps that remained in component logs and Grafana metrics to keep narrowing the incident window. The sequence placed the guest-visible read and write slowdown and falling I/O volume first, followed by etcd coordination delays, kube-apiserver deadlines, and leader-election loss. That order was consistent with a lower-layer stall propagating upward, but the reconstruction alone did not prove causality.
| Status | Statement | Evidence boundary |
|---|---|---|
| Known | Controllers exited after leader-election renewal failed. | Previous-process evidence recorded the termination path. |
| Known | kube-apiserver and etcd experienced delayed operations in the incident window. | Their logs described deadlines, slow coordination, and leadership disruption. |
| Known | Several VM roles showed rising read and write latency while I/O volume fell in the same monitoring window. | The pattern was not confined to a busy etcd member. |
| Leading inference | A shared lower-layer stall plausibly delayed etcd, kube-apiserver, and leader renewal. | The dependency order and cross-role correlation fit this explanation better than an isolated controller crash. |
| Still open | The exact mechanism: host scheduling, VM pause, network, virtual block path, backing storage, or another shared dependency. | The decisive evidence belonged to infrastructure systems outside the Kubernetes access boundary. |
The surviving timestamps and monitoring intervals were sufficient to narrow the window and reconstruct a plausible sequence, but not to turn that sequence into a final infrastructure root cause.
This was the strongest statement our layer could support. Calling it a storage root cause would have hidden the remaining alternatives. Calling it only a Kubernetes restart would have ignored the cross-layer evidence. The report needed to preserve both the useful inference and the uncertainty around it.
05 / HAND OFF THE MISSING PROOF
In a split-ownership environment, the report was the action I could take.
Air-gapped access did not grant infrastructure authority. I could inspect Kubernetes behavior and guest-visible metrics during the available access window. I could not inspect every hypervisor, virtual-disk queue, backing-storage path, network event, or maintenance action, and I could not make changes in those systems.
The service path was healthy, and the strongest unanswered questions now sat below Kubernetes. Changing cluster settings or restarting components without evidence would not have tested the leading hypothesis. It would only have added another variable, so I left the healthy cluster alone.
I organized the result as a report for the project manager to pass to the dedicated infrastructure team. It did not say that the platform was simply slow. It separated the current healthy state from the incident window, ordered the component evidence, stated why the leading hypothesis had changed, and kept the unresolved alternatives visible.
- Did the affected guests share a compute host, storage path, or another failure domain during the incident window?
- Did the hypervisor or VM scheduler record pauses or scheduling pressure that the guests could not see?
- Did the virtual block layer or backing storage record latency, queueing, failover, or maintenance at the same time?
- Could a shared network or infrastructure path explain the cross-role pattern better than the leading stall hypothesis?
Only the infrastructure team had the telemetry needed to answer those questions. The investigation and follow-up remain in progress, so this article ends without a final root cause or a completed remediation. That is the current state of the work, not a gap to fill with a stronger-sounding conclusion.
I did not leave this investigation with a fix I could claim. I left with a clearer separation between three states: the service can look healthy now, the observed failure path can be well supported, and the initiating mechanism can still be open. In a constrained environment, keeping those statements separate is part of the engineering work.