[opensci small-binder] Notebook nodegroups are not available
| Field | Value |
|---|---|
| Impact Time | Dec 16 at 15:55 to Dec 19 at 13:31 |
| Duration | 2d 21h 36m 9s |
What Happened¶
The opensci small-binder cluster didn’t have any notebook nodegroups available, so users could not spawn servers.
Resolution¶
Deleting and recreating the nodepools fixed it.
Where We Got Lucky¶
We had people working so close to the winter PTO break.
What Went Well¶
The fix was a quick one.
What Didn’t Go So Well¶
Because this happened during winter time off this meant that we fixed the issue for this cluster, but didn’t go to the bottom of the problem.
The problem was that we reached the maximum number of nodegroups per clustee. This caused another outage later on.
Action Items¶
Timeline¶
Dec 19, 2025¶
| Time | Event |
|---|---|
| 1:05 PM | Engineer notices that scale-ups aren’t happening Georgiana Dolocan #opensci-small-binder-missing-nb-nodegroups-dec2025 scale-ups aren’t triggering |
| 1:06 PM | Engineer notices missing node groups Georgiana Dolocan #opensci-small-binder-missing-nb-nodegroups-dec2025 and looking at the nodegroups for small-binder I can only see dask ones for some reason. |
| 1:06 PM | Engineer recreates the nodegroups Georgiana Dolocan #opensci-small-binder-missing-nb-nodegroups-dec2025 I’ll try recreating |
| 1:28 PM | Problem is resolved Georgiana Dolocan #opensci-small-binder-missing-nb-nodegroups-dec2025 Ok, this fixed it 1:31 PM [FIRING:1] Two servers failed to start in the last 30m opensci small-binder (immediate action needed) |