Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Incident Reports

This is a public record of incidents for 2i2c managed cloud services[1].

Why We Share These

Transparency and learning. We believe openly sharing what goes wrong, and how we fix it, makes everyone better. These reports help us improve our operations and help others learn from our experiences.

Each report includes:

  • What happened and why

  • How we resolved it

  • What went well and what didn’t

  • Action items for improvement

All Reports

DateReportDuration
2026-02-12Resource exhaustion on us-west2 for HHMI1d 14h 48m 54s
2026-02-11[dubois:ephemeral] Unable to start servers13h 58m
2026-01-13Core node restarts on LEAP17h 52m 20s
2025-11-26EarthScope InvestigationUnknown
2025-10-17Enforce-nfs-quota pod restarts on victor19m 23s
2025-10-16OOM kill of enforce-xfs-quota on neurohackademy1h 14m
2025-10-16Invalid configuration data breaks enforce-nfs-quota container24m 23s
2025-10-15Starting Server of users with 2i2c admin emails from admin panel fails on earthscope17m 6s
2025-10-08Oceanhackweek hub URL returned 503 response20m
2025-10-03UToronto: Users who have never logged in before can’t start servers23m 24s
2025-08-29Students experienced issues when a number of them were trying to login to the hub at the same time during start of class.4d 1h 30m
2025-08-26LEAP out of GPU quota2h 13m 22s
2025-08-20JupyterHub at temple.2i2c.cloud unreachable13m 46s
2025-08-12LEAP hub outageUnknown
2025-07-21Openscapes hub pods dying post-stress-testingUnknown
2025-07-11CloudBank health check fail plus GroupExporter pod restartsUnknown
2025-05-12Earthscope resource provisioning issueUnknown
2023-02-012023-02-01 Heavy use of dask-gateway induced critical pod evictionsUnknown
2023-01-232023-01-23 Upgrading ingress-nginx caused brief outage in carbonplanUnknown
2022-11-14LIS hub cannot scale6h 30m
2022-10-31UToronto Hub Login failing with 500 errors2h 33m
2022-09-06UToronto Hub is throwing 500 errors when users try to login5h 23m
2022-08-30UToronto JupyterHub not accessible58m
2020-10-282020-08-28 - Memory overload on WER clusterUnknown

Learn More

Footnotes

  1. Reports are automatically generated from PagerDuty postmortems and published via GitHub Actions.

    ↩
Incident reports
2020-08-28 - Memory overload on WER cluster