Instructions for setting up an AWS S3 bucket for your project

This set of instructions will walk through how to setup an AWS S3 bucket for a specific project and how to configure that bucket to allow all members of the project team to have access.

Create an AWS account and S3 bucket¶

The first step is to create an AWS account that will be billed to your particular project. This can be done using these instructions.

Create AWS S3 bucket¶

Within your new AWS account, create an new S3 bucket:

1. Open the AWS S3 console (https://console.aws.amazon.com/s3/)
2. From the navigation pane, choose Buckets
3. Choose Create bucket
4. Name the bucket and select us-west-2 for the region
5. Leave all other default options
6. Click Create Bucket

Create a user¶

Within the same AWS account, create a new IAM user:

1. On the AWS Console Home page, select the IAM service
2. In the navigation pane, select Users and then select Add users
3. Name the user and click Next
4. Attach policies directly
5. Do not select any policies
6. Click Next
7. Create user

Once the user has been created, find the user’s ARN and copy it.

Now, create access keys for this user:

1. Select Users and click the user that you created
2. Open the Security Credentials tab
3. Create access key
4. Select Command Line Interface (CLI)
5. Check the box to agree to the recommendation and click Next
6. Leave the tag blank and click Create access key
7. IMPORTANT: Copy the access key and the secret access key. This will be used later.

Create the bucket policy¶

Configure a policy for this S3 bucket that will allow the newly created user to access it.

1. Open the AWS S3 console (https://console.aws.amazon.com/s3/)
2. From the navigation pane, choose Buckets
3. Select the new S3 bucket that you created
4. Open the Permissions tab
5. Add the following bucket policy, replacing USER_ARN with the ARN that you copied above and BUCKET_ARN with the bucket ARN, found on the Edit bucket policy page on the AWS console:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListBucket",
            "Effect": "Allow",
            "Principal": {
                "AWS": "USER_ARN"
            },
            "Action": "s3:ListBucket",
            "Resource": "BUCKET_ARN"
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Principal": {
                "AWS": "USER_ARN"
            },
            "Action": "s3:*Object",
            "Resource": "BUCKET_ARN/*"
        }
    ]
}

Configure local AWS profile¶

To access the bucket from a “local” computer (for example, a local computer or a JupyterHub that is running on AWS), an AWS profile must be configured with the access keys to the new S3 bucket.

This can be done manually but the easiest way to do it is by using AWS Command Line Interface (CLI).

On the local computer:

1. Open a terminal
2. Install AWS CLI (if using the conda package manager, you can install like this: conda install -c conda-forge awscli)
3. Configure the AWS profile with the keys from above and give your profile a new profile name:

• aws configure --profile PROFILE_NAME
• Enter the access key and secret access key
• Enter us-west-2 for the region and json for format

In the code examples below, the profile name is icesat2. Replace this with the profile name that you used.

Example code¶

Below is code that can be used to read from and write to the S3 bucket in order to test that the bucket and local AWS profile have been configured correctly. Below each cell, there is expected output.

In the examples below, we are accessing a bucket called gris-outlet-glacier-seasonality-icesat2. Replace this name with the name of the S3 bucket that you have created.

Reading from the S3 bucket¶

Example: ls bucket using s3fs¶

Example: open HDF5 file using xarray¶

Example: read a geotiff using rasterio¶

Writing to the S3 bucket¶