How solving a disk usage bug for one community benefitted all communities

Scott Henderson noticed core dump files appearing on home directories that were taking up space when a program crashes in one of the hubs we run for the CryoCloud community. While core dump files can be useful in some circumstances, they are almost never useful on a hub. They can also be huge, costing real cloud spend. Ian Caroll noticed the increase in disk usage happening again on both the CryoCloud and the openscapes hubs as well. Ian pointed out a fix, but it would need to be adapted to run on kubernetes. 2i2c is a small organization, and at that moment we did not have the capacity to do that work.

Separately, Alex Mandel from Development Seed, working for the NASA VEDA project also noticed this in their hubs in late 2025, as part of their efforts to audit home directory usage and control costs. They were able to build on top of the work from the other communities we supported, and with help from Chuck Daniels and Tarashish Mishra (also from Development Seed), they were able to:

1. Clean up all core dumps in their communities, and reclaim 171G of space
2. Operationalize the kubernetes setup needed to disable core dumps on an ongoing basis.

We were able to roll this out to all our communities, and saved communities a lot of space. In particular, the CryoCloud community was saved 717G, which translates to a lot of money! Also poignant that they were the first community to notice the problem.

We’re also working on contributing this upstream to zero to jupyterhub, as this is valuble to everyone running a JupyterHub, not just us.

2i2c’s Value #

This is a loop of value that we intentionally try to encourage and bring about at 2i2c:

1. Community A notices a problem
2. We recognize this is a general problem affecting a lot of communities, but just observed in Community A
3. We informally put together a collection of communities that can contribute different things to solve the problem - observations, engineering time, validation, etc.
4. We solve the problem for everyone, including the folks who didn’t know they had a problem!

This is a generalization of the overall loop of open source software adding value to the world, and I’m excited to see it play out in a way that’s organizationally supported by 2i2c.

Acknowledgements #

• Thanks to CryoCloud, Development Seed, and CryoCloud for helping with identifying and making this improvement!
• This work was funded in-part by NASA VEDA