TLS certificates expired for two 2i2c hubs
| Field | Value |
|---|---|
| Impact Time | Apr 15 at 12:50 to Apr 15 at 14:17 |
| Duration | 1h 26m 47s |
Overview¶
Following migration to nginx-ingress, broken certificate renewals led to the expiry of TLS certificates on two hubs
What Happened¶
We discovered that TLS certificates had already expired. Although we knew of this risk, the script used to identify the deadline for renewals had a small parsing bug that failed to catch these hubs.
Resolution¶
2i2c patched the Certificate CRDs and manually re-issued the CertificateRequest objects, triggering the certificates to be reissued.
Where We Got Lucky¶
We were already working on the wider problem of certificate renewals, and thus were checking for certificate problems.
What Went Well¶
We knew why certificates had not been reissued, and had a clear process to resolve the matter
What Didn’t Go So Well¶
Our previous checks to identify at-risk clusters was faulty