Connect AWS S3 as external library

• Automated SQS & SNS Queue creation with an existing S3 Bucket 
• Creation of the external S3 Library in Schedule
• Manual creation of AWS SQS & SNS Configuration

Automated SQS & SNS Queue creation with an existing S3 Bucket 

To connect the LTN Schedule with your AWS Account/S3 Bucket, we recommend to create a new IAM-account just for Schedule, these permissions are needed if schedule creates the SQS and SNS Queues for your S3 Bucket.

This account must have read-write-access to use Simple Notification Service (SNS) and Simple Queue Service (SQS) along to S3. The Schedule needs access to SNS and SQS to get notified about changes in the S3-Bucket, otherwise it would have to scan the whole bucket periodically, which costs a lot of time and would result in changes not being visible in a timely manner. 

For the current implementation we advice the following additional IAM Policy for the Schedule IAM User. It is possible to restrict s3 access to certain folders by restricting access to certain resources for the IAM user.

1. Create a AWS IAM User with the following Policy to list all S3 Buckets in your AWS Account and create the SQS and SNS Queue. Please note that this requires Programmatic Access Access for the user e.g AWS cli credentials

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3access",
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:PutBucketNotification",
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket-name>",
                "arn:aws:s3:::<bucket-name>/*"
            ]
        },
    {
        "Sid": "SNSandSQS",
            "Effect": "Allow",
            "Action": [
                "sns:ListSubscriptionsByTopic",
                "sns:Publish",
                "sns:GetTopicAttributes",
                "sns:DeleteTopic",
                "sns:CreateTopic",
                "sns:Unsubscribe",
                "sns:SetTopicAttributes",
                "sns:Subscribe",
                "sqs:DeleteMessage",
                "sqs:ListQueues",
                "sqs:RemovePermission",
                "sqs:GetQueueUrl",
                "sqs:AddPermission",
                "sqs:ReceiveMessage",
                "sqs:DeleteQueue",
                "sqs:SendMessage",
                "sqs:GetQueueAttributes",
                "sqs:CreateQueue",
                "sqs:SetQueueAttributes",
                "s3:ListAllMyBuckets"
            ],
            "Resource": "*"
        }
    ]
}

With this policy the Schedule is able to create a new SNS Queue with a attached SQS Topic, each S3 Bucket gets his own SQS and SNS Queue.

Creation of the external S3 Library in Schedule

Login to your Schedule instance 

1. Click on the extend Button at your Media Drawer.

2. Click on Manage Libraries.

3. Click on "+" and add a New Amazon S3 Library.

4. Add your IAM Credentials and click on Done.

5. After that you will see this: Click on edit and choose your bucket and click on Done.

6. Your bucket is added.

7. Files

New Files will show up inside the Library after a few seconds.

Manual creation of AWS SQS & SNS Configuration

This article describes how to create and attach the AWS SQS Queue to an AWS S3 Bucket with a user creation, that can be used within a Schedule library. The following description assumes the S3 bucket and the SQS Queue are in the same AWS Region. This is the AWS Article Walkthrough: Configuring a bucket for notifications (SNS topic or SQS queue) with a IAM User who can read the SQS Queue 

The top rundown is as follows

1. create a S3 bucket
2. create a SQS and SNS queue and configure it for S3 events
3. add the Access policy so that the s3 bucket can publish SNS events
4. attach the SNS Topic to the S3 bucket
5. create a AWS IAM user with permissions to the bucket and the SQS/SNS
6. adding the library in Schedule itself

1. S3 bucket creation

Configure the properties as shown on the screenshot and leave the tab open. We will need this soon.

Note: The configuration focuses on the most important settings, not going into details about encryption settings/cloudwatch metrics/versioning object logging and so on.

2. SNS Topic creation

Create a new Topic and add a new subscription, choose Amazon SQS as Protocol and paste the SQS ARN as the endpoint and click create, next we update the Access Policy for the Topic.

We only need to add a new Statement in the context and save the changes.

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      .....default permissions...
    },
    {
      "Sid": "LTNScheduleGeneratedSNSPolicy_YOUR_BUCKET",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "sns:Publish",
      "Resource": "YOUR_SNS_ARN",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:*:*:YOUR_BUCKET"
        }
      }
    }
  ]
}

 

3. SQS Queue creation

Head over to the SQS Management interface in the AWS console and create a simple Standard Queue.

In the Access Policy section, chose Advanced with the following json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "1",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "sqs:SendMessage",
      "Resource": "YOUR_SQS_ARN",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "YOUR_SNS_ARN"
        }
      }
    }
  ]
}

 Click "Create Queue". Now go to the Access Policy and replace the SQS Arn and the SNS ARN.

4. S3 Event configuration

With the configured SNS/SQS Topic we can now configure events for the S3 Bucket, to do so go to the "Advanced settings" of your S3 Bucket and click on Events › "Add notification".

Define a custom name and select  "All object create events", if all your media files are in a sub folder also define a Prefix. Under "Send To" select the SNS Topic with the configured Queue we've created and click on Save.

5. AWS IAM User 

With the events set up we can now create the IAM user with the permissions required.

Create a new user with Programmatic access with "Add inline policy", select json.

The user needs read permissions on the SQS Queue and the GetObject* and GetBucket* on the S3 Bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:PutBucketNotification",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:sqs:......",
                "arn:aws:s3:::.....",
                "arn:aws:s3:::...../*"
            ]
        }
    ]
}

6. Schedule Configuration

in the library configuration, set the Access key for the user and the Secret key for the password, and copy the bucket and SQS ARN into the respective fields and click on save.