Earthscope GeoLab Login Failure due to mismatched redirect URL

Field	Value
Impact Time	Apr 6 at 08:30 to Apr 6 at 23:36
Duration	15h 6m 11s

Overview¶

The earthscope community reported total login failures on their hub via our support system: https://2i2c.freshdesk.com/a/tickets/5413. This was fixed by explicitly setting oauth redirect config instead of relying on guessing via HTTP headers: 2i2c-org/infrastructure#8053

Resolution¶

We no longer guess the oauth_callback_url, instead explicitly setting it https://github.com/2i2c-org/infrastructure/pull/8053
We verify that no other hub was relying on this guessing behavior

Where We Got Lucky¶

An engineer in the US timezone was available
The community found the issue and escalated it before they had a workshop

What Went Well¶

Our phased rollout means we have been able to handle issues one at a time as they come, instead of being overwhelmed with them in one go

What Didn’t Go So Well¶

The community spotted the issue, and our automation did not catch this at all.

Action Items¶

2i2c-org/infrastructure#8157

Timeline¶

Time	Event
8:16AM	2i2c-org/infrastructure#8049 was merged Migrates from ingress-nginx to nginx-ingress, as part of a phased rollout
10:30PM	https://2i2c.freshdesk.com/a/tickets/5413 arrives, with earthscope reporting login failures to us
10:52PM	Incident acknowledged and investigated
11:03PM	2i2c-org/infrastructure#8053 is deployed manually, fixing the issue OAuthenticator is guessing the callback URL, and we suspect tornado can’t quite get it right with nginx-ingress. Explicitly setting the callback URL fixed the issue for GeoLab, and we verified that no other hub using OAuth is relying on the guessing behavior
11:06PM	Triggered by Yuvi Panda through the website. Description: Earthscope GeoLab Login Failures (View Message) INCIDENT #2080 Earthscope GeoLab Login Failures
11:36PM	Resolved by Yuvi Panda through the website. INCIDENT #2080 Earthscope GeoLab Login Failures

Apr 7, 2026¶

Time Event

12:00 AM nginx-ingress only sets X-Forwarded-Proto, while ingress-nginx set X-Forwarded-Proto and X-Scheme We compare logs from a past hub that was on ingress-nginx and the earthscope hub, obtainable by triggering any error on the hub. We notice that nginx-ingress sets both XScheme and X-Forwarded-Proto, while ingress-nginx only sets X-Forwarded-Proto. However, X-Forwarded-Proto is appended to by every hop, and so by the time it reaches the hub, its value is https,http - it’s appended to by configurable-http-proxy, using this code https://github.com/http-party/node-http-proxy/blob/9b96cd725127a024dabebec6c7ea8c807272223d/lib/http-proxy/passes/webincoming.js#L78. configurable-http-proxy did not do this for X-Scheme, so it passed through unchanged. Tornado prefers X-Scheme over X-Forwarded-Proto (https://github.com/tornadoweb/) tornado/blob/0ca3c5f8279d402b245718d16522bc18a8f5d958/tornado/ httpserver.py#L352) so guessing the URL worked fine when X-Scheme was set to https. But when X-Scheme isn’t set, and X-Forwarded-Proto is https,http, tornado interprets that as http (tornado/httpserver.py). This breaks the guessing.

Time	Event
12:00 AM	nginx-ingress only sets X-Forwarded-Proto, while ingress-nginx set X-Forwarded-Proto and X-Scheme We compare logs from a past hub that was on ingress-nginx and the earthscope hub, obtainable by triggering any error on the hub. We notice that nginx-ingress sets both XScheme and X-Forwarded-Proto, while ingress-nginx only sets X-Forwarded-Proto. However, X-Forwarded-Proto is appended to by every hop, and so by the time it reaches the hub, its value is `https,http` - it’s appended to by configurable-http-proxy, using this code https://github.com/http-party/node-http-proxy/blob/9b96cd725127a024dabebec6c7ea8c807272223d/lib/http-proxy/passes/webincoming.js#L78. configurable-http-proxy did not do this for X-Scheme, so it passed through unchanged. Tornado prefers X-Scheme over X-Forwarded-Proto (https://github.com/tornadoweb/) tornado/blob/0ca3c5f8279d402b245718d16522bc18a8f5d958/tornado/ httpserver.py#L352) so guessing the URL worked fine when X-Scheme was set to https. But when X-Scheme isn’t set, and X-Forwarded-Proto is `https,http`, tornado interprets that as http (tornado/httpserver.py). This breaks the guessing.