LEAP hub and Grafana are down
| Field | Value |
|---|---|
| Impact Time | Sep 20 at 17:18 to Sep 21 at 15:57 |
| Duration | 22h 39m 49s |
Overview¶
The hub and Grafana were not reachable because of a core node that ran out of TCP memory. Even though the hub and grafana pods were running without issues, the pods on the problematic nodes were failing, including kube-dns, which made the hub and Grafana unreachable networking wise.
The system eventually self-healed and the hub and Grafana showed up again.
What Happened¶
A core node ran out of TCP memory, which made the kube-dns pod unable to resolve DNS queries.
The pod tried to restart a few times but failed. During this time frame (~30m) the hub and Grafana were not reachable.
According to logs and pods age, the hub and the prometheus never stopped running during this time frame.
Resolution¶
It wasn’t until the node was killed and kube-dns was scheduled on another node that the issue was fixed.
Where We Got Lucky¶
An user reported the problem and the system self-healed in about 30m.
What Went Well¶
The system self-healed in about 30m and we were able to take the time to find the issue.
What Didn’t Go So Well¶
The community reported the issue to us.
Logs exporter showed the TCP OOM error as an INFO message, which made it really hard to spot.
Action Items¶
Timeline¶
Sep 20, 2023¶
| Time | Event |
|---|---|
| 5:19 PM | Cluster events showed messages about node/gke-leap-cluster-core-pool-1592e1d3-gnks being healthy 目 Running k get events shows the following events related to node/gke-leap-clustercore-pool-1592e1d3-gnks: - 113s Normal NodeNotReady - 2m48s Normal NodeHasSufficientMemory - 2m48s Normal NodeHasNoDiskPressure - 2m48s Normal NodeHasSufficientPID - 2m48s Normal NodeReady |
| 5:31 PM | By this time, the hub and Grafana became available again and the outage ended P Georgiana Dolocan #managed_jupyt_inc_262 Ok, for some reason the hub and grafana are up again :confused: I’m poking around the console to figure out what happened Georgiana Dolocan #managed_jupyt_inc_262 I’m seeing bunch of error in the support-ingress-nginx-controller pod, Also, the support-ingress-nginx-controller and the support-grafana pods seem to have been restarted around the time the hub and grafana came up again. Georgiana Dolocan #managed_jupyt_inc_262 Also, in grafana there’s a 2:30h gap in data Georgiana Dolocan #managed_jupyt_inc_262 My only explanation so far is that the something happened with node gke-leap-clustercore-pool-1592e1d3-gnks which made all of the pods on it restart before the node died and the pods were rescheduled on a new one: Georgiana Dolocan #managed_jupyt_inc_262 I’ve seen a weird message about the reason why this node poll was deleted "Deleting node gke-leap-cluster-core-pool-1592e1d3-gnks because it does not exist in the cloud provider". Urgency changed from high to low by Georgiana. INCIDENT #262 LEAP hub and Grafana are down Georgiana Dolocan #managed_jupyt_inc_262 Ok :slightly_smiling_face: I think I will logout and try to wind off. My conclusion atm is that the hub was never down, just unreacheable because of the kube-dns pod restarting on another node. Also, the time it took to figure out that the old 8:09 PM |