Data and Filesystem

This section describes how files and data storage are handled in 2i2c Hubs.

Quick recommendations

Some quick recommendations for how to handle data:

• Use your home directory to store code, notebooks, and small data files (<1 GB) for personal use

• Use cloud object storage to store larger datasets and to share data across your team

• Consider whether your project would benefit from other cloud-native data storage solutions such as a database, data warehouse, or data lake

For more information, see the sections below.

• Filesystem and user directory
  • Your Home Directory
    • Modify your bash profile
  • The shared Directory
  • The /tmp Directory
• Move code in and out of the hub with GitHub
  • Using gh-scoped-creds
  • Enabling gh-scoped-creds for your hub
• Share data files with your users
  • The shared directory
  • The allusers directory - available upon request
  • A workflow for sharing datasets
• Cloud Object Storage
  • Cloud-Native Formats
  • Working with Object Storage
    • Reading Data
    • Writing Data
  • Scratch Bucket
  • Data Catalogs

References and attribution

Much of this material was adapted from the Pangeo Cloud User Guide.