Degraded performance of EarthScope prod hub

Overview¶

During a cloud computing and parallelization for seismologists workshop, attendees experienced degraded server performance following the onset of high volume disk writing

What Happened¶

~30 attendees of the workshop ran parallelised analysis routines. These routines performed high-volume IO to the home directory storage. Subsequently, the EBS disk backing the central NFS server reached its provisioned throughput limit, leading to iowait on the core node CPU. This led to user servers experiencing severe delays in FS operations on home directories, which manifested as degraded user experience in the JupyterLab interface and kernel execution.

Resolution¶

Once throughput demand dropped below the provisioned value, the iowait on the core node dropped to a nominal value.

What Went Well¶

We had an alert to detect large iowait incidences.

What Didn’t Go So Well¶

The iowait saturation alert took 30m to trigger a PagerDuty incident from the onset of the incident.

The incident occurred outside of regular business hours (in the late evening) for the on-staff engineers, meaning that they didn’t respond in realtime.

The iowait alerts were not considered alarming when retrospectively reviewing the alerts the following day.

Action Items¶

• 2i2c-org/infrastructure#8128

• 2i2c-org/infrastructure#8129

• 2i2c-org/infrastructure#8130

• 2i2c-org/infrastructure#8131

• 2i2c-org/infrastructure#8132

• 2i2c-org/infrastructure#6249

• 2i2c-org/infrastructure#8133

• 2i2c-org/infrastructure#8221

Timeline¶

Apr 14, 2026¶

Apr 15, 2026¶