How to Securely Share AWS S3 Files

March 02, 2022



AWS S3 is one of the most used and popular AWS services today. Everyone needs a repository service for file storage, backup, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage. Storage needs are increasing every day and for this reason; building and maintaining your system becomes difficult and complex. AWS S3 is the best fit for these needs because it provides scalability, high availability, low latency, and low cost. Also, its usage is very easy. It’s possible to create our AWS S3 bucket and upload any files that we want. It can also be integrated with other AWS services such as AWS Lambda, AWS API Gateway, AWS CloudFront, etc.

This image shows AWS S3 uses cases.

While it becomes attractive for developers, it also becomes an exciting storage place for attackers. Data leaks are becoming extremely critical for companies if the necessary AWS S3 configurations are not made. There are some security points that need to be considered when you’re creating your own AWS S3 bucket:

  • Always consider blocking public access first.
  • Use AWS S3 encryption.
  • Limit your IAM permissions to AWS S3 buckets. Always follow at least the privilege principle.
  • Enforce HTTPS to prevent attacks like MITM (Man in the Middle).
  • Enhance S3 security using logging.

In this blog post, we will talk about a real-world use case for sharing AWS S3 files securely. We will provide the architecture and configurations for all steps that you will need. Let’s start!

Scenario:

A company wants to share invoice reports weekly. Here are the requirements:

  • After a week, reports cannot be reachable for anyone.
  • Reports contain critical data like financial details, so they should not be accessible from a public URL.
  • The process should be simple as the reporters are not competent in coding.
  • The company manager does not want to pay much for the process.

Solution: AWS S3 pre-signed URL generation for every upload!

Let’s review the architecture together:

This image shows the architecture of the use case scenario.

  1. Firstly, we need to create a private AWS S3 bucket for uploading reports. Then we have to create a new API Gateway for uploading reports to the AWS S3 bucket with API requests. So we will be able to upload reports with the API requests.

This image shows S3 bucket permissions.

Our AWS API Gateway should look like below:

This image shows API Gateway Overview.

We need to get invoke URL from the created AWS API Gateway:

This image shows AWS API Gateway Invoke URL.

Note: Keep this invoke URL for the next steps. You will put the object in the AWS S3 bucket.

Send the API request to invoke URL for upload reports to AWS S3 (We’re using Postman for this) :

This image shows API Request.

API request body:

This image shows API Request’s Body.

Note: This API is accessible for anyone. Attackers love unauthorized access. To prevent this, we need to do allow only specific IP addresses access to our Amazon API Gateway. These IP addresses can be VPN IPs of the company. You can use resource policies for this.

  1. We’re checking the reports are uploaded to AWS S3 successfully.

This image shows the uploaded report.

  1. We have uploaded reports via API request to create an AWS S3 bucket. Now we need an AWS Lambda function that invokes uploaded reports. This function will generate a pre-signed URL that will expire after 7 days and send this to employees that are defined on AWS SNS. Employees need to be subscribed to the AWS SNS topic before this operation.

Note: All S3 objects are private by default. They can only be accessed by the object owner. By creating a pre-signed URL, the object owner can share objects with others. To create a pre-signed URL, you should use your own credentials and grant time-limited permission to download the objects. The maximum expiration time for a pre-signed URL is one week from the time of creation and there is no way to have a pre-signed URL without expiry time.

You can use the Lambda function code below:

import json
import boto3
def lambda_handler(event, context):
    s3_bucket_name = str(event['Records'][0]['s3']['bucket']['name'])
    s3_report_key = str(event['Records'][0]['s3']['object']['key'])
    s3_client = boto3.client('s3')
    report_presigned_url = s3_client.generate_presigned_url('get_object', Params={'Bucket': s3_bucket_name, 'Key': s3_report_key}, ExpiresIn=604800)
    MY_SNS_TOPIC_ARN = "{SNS_topic_ARN}"
    sns_client = boto3.client('sns')
    sns_client.publish(
        TopicArn=MY_SNS_TOPIC_ARN,
        Subject = 'Reports Presigned URLs',
        Message="Your Reports are ready:\n%s" %report_presigned_url)
    print('success')
    return {'message': 'it works'}

We need to attach the required policies to the AWS Lambda. For test purposes, full access policies are attached. If you’re using this code in production, you need to create your own policies with at least a privilege principle.

This image shows the required permissions for AWS Lambda.

  1. Now, we should add the event notification that will trigger the Lambda when the report is uploaded to the s3 bucket. We need to navigate “Amazon S3 → {bucket_name} → Properties → Event notifications” and create a new event notification:

This image shows the event notification configuration.

We need to select the Lambda function’s ARN that we created before:

This image shows the event notification configuration.

  1. We should also check reports on the S3 bucket and delete reports whose creation date is more than 7 days. For this, we will create a lambda function and make it trigger every 7 days from AWS CloudWatch. You can use the Lambda function code below:
import json
import boto3
import time
from datetime import datetime, timedelta
def lambda_handler(event, context):
    old_files = []
    s3 = boto3.client('s3')
    try:
        files = s3.list_objects_v2(Bucket={bucket_name})['Contents']
        old_files = [{'Key': file['Key']} for file in files if (file['LastModified'].strftime('%Y-%m-%d')) < ((datetime.now() - timedelta(days=7)).strftime('%Y-%m-%d'))]
        if old_files:
            s3.delete_objects(Bucket={bucket_name}, Delete={'Objects': old_files})
			return {
            'statusCode': 200,
            'body': "Expired Objects Deleted!"
            }
    except:
        return {
            'statusCode': 400,
            'body': "No Expired Object to Delete!"
            }

When we create the CloudWatch Event Rule, we need to select the event source and targets.

This image shows CloudWatch Event Rule.

  1. All processes are done and ready to use. We can upload reports with the API request. After we send a request to our API successfully, we’ll be able to get a pre-signed URL in an e-mail which will look like this:

This image shows the incoming mail.

When we click on the pre-signed URL, we will be able to download the report we have uploaded with the API request.

This image shows the downloaded file by clicking the pre-signed URL.

Also, the event rule we created in CloudWatch will be checking our S3 bucket weekly and will delete those that have been there for more than 1 week.

In this blog, we’ve summarized what AWS S3 is. We also prepared a real-world use case for sharing AWS S3 files securely. We provided the architecture and configurations for all steps that you need. We hope you enjoyed it!

Check out our Cloud Security services to stay secure in the cloud!