Starting Server of users with 2i2c admin emails from admin panel fails on earthscope
| Field | Value |
|---|---|
| Impact Time | Oct 15 at 10:28 to Oct 15 at 10:45 |
| Duration | 17m 6s |
Overview¶
An authorized admin user on the earthscope hub attempted to start a server for 2i2c staff accounts via the JupyterHub admin panel. Because these accounts were set up for Google Auth emails, the server startup failed, triggering an alert for server startup failure.
Resolution¶
We manually removed the 2i2c staff accounts with google emails, and merged a PR to stop adding them 2i2corg
What Went Well¶
There were no user facing issues
Our automated alerting setup caught it immediately
Action Items¶
Make sure that custom auth hubs don’t have unrelated 2i2c staff engineer accounts added, as this is confusing in the admin panel 2i2c
-org /infrastructure #6945
Timeline¶
| Time | Event |
|---|---|
| 10:26 AM | Authorized admin user logs into the hub, and accesses the admin panel. They try to start server of a 2i2c user who is listed as admin. This server start fails, because the earthscope hub uses auth0 auth but the admin accounts were added as google email addresses |
| 10:28AM | Triggered through the API. Description: [FIRING:1] Server Startup Failed earthscope prod (same day action needed) (View Message) INCIDENT #1558 Starting Server of users with 2i2c admin emails from admin panel fails on earthscope |
| 10:37AM | 2i2c |
| 10:40AM | Existing 2i2c staff email based jupyterhub admin accounts are manually removed through the JupyterHub interface |
| 10:45AM | Resolved by Yuvi Panda through the website. INCIDENT #1558 Starting Server of users with 2i2c admin emails from admin panel fails on earthscope |