BinderHub URL responds Bad Gateway for Project Pythia
| Field | Value |
|---|---|
| Impact Time | Mar 9 at 16:39 to Mar 10 at 00:58 |
| Duration | 8h 19m 9s |
Overview¶
A community representative reported 502 errors from the Project Pythia BinderHub URL. This was caused by an update to the ingress that was made when the cluster was recently recreated and the hub redeployed.
What Happened¶
Errors in the nginx-ingress controller led to the reverse proxy being unable to communicate with the binderhub service. This resulted in 502 errors.
We also identified websocket failures between the frontend and the user server
Resolution¶
The upstream chart was fixed and temporarily deployed to the Binder hub, and the new ingress patched to upgrade websocket connections properly.
Where We Got Lucky¶
We have been communicating with the Pythia community in a high-bandwidth manner, so it was quicker to identify and resolve.
What Went Well¶
The diagnosis process was fairly quick!
What Didn’t Go So Well¶
We got an automated alert about this 20min after the community representative reached out to us to let us know
Action Items¶
Timeline¶
| Time | Event |
|---|---|
| 4:39 PM | Incident reported by community representative, both 502 and websocket failures Description: BinderHub URL responds Bad Gateway for Project Pythia (View Message) BinderHub URL responds Bad Gateway for Project Pythia |
| 5:21 PM | Engineer suspects the nginx-ingress is root cause for websocket errors Angus Hollands #jupyterhub_u_inc_1942 OK, it’s all websocket handlers - I’m going to suspect the nginx-ingres |
| 6:02 PM | Engineer reports suspicion of missing upgrade annotation Angus Hollands #jupyterhub_u_inc_1942 My hunch is that this is probably a failure of url rewriting with the websocket scheme on the rewrite ingress annotations. I’ll see if I can log or test that |
| 7:26 PM | Engineer reports that reverting to the old ingress fixes both issues Angus Hollands #jupyterhub_u_inc_1942 I reverted the ingress and things are working -- I suspect this is the problem |
| 9:09 PM | Engineer observes that setting an NGINX snippet for websocket upgrade fixes websocket errors Angus Hollands #jupyterhub_u_inc_1942 Cool, setting a custom snippet to enable ws upgrade fixes this. |
Mar 10, 2026¶
| Time | Event |
|---|---|
| 12:58 AM | 502 errors resolved by upstream patch to binderhub chart Note added by Angus. Resolution Note: The incident "BinderHub URL responds Bad Gateway for Project Pythia " has been resolved. BinderHub URL responds Bad Gateway for Project Pythia |