[earthscope, prod] Servers unable to launch

Overview¶

Users could not launch servers for 1h14m.

What Happened¶

Ongoing improvements to the usage quota system introduced a bug into the server spawn experience that was not caught by Continuous Integration (CI).

Resolution¶

The most recent update to the hub image was reverted.

Where We Got Lucky¶

An engineer was available outside of working hours to resolve the issue.

What Went Well¶

A team member was able to quickly flag the issue following a report from FreshDesk.

What Didn’t Go So Well¶

No engineer was able to respond to the automated alerts that were triggered prior to the FreshDesk report from a community member.

Action Items¶

• prioritize https://github.com/2i2c-org/infrastructure/issues/7648 to improve CI that would have flagged this issue immediately

• clearly define internal and external policy towards out-of-hours support as suggested in https://github.com/2i2c-org/infrastructure/issues/8132