Queuing services (SQS, Kafka)

AWS SQS (Simple Queue Service)

AWS SQS is a managed queuing system allowing you to get messages delivered straight into your existing queue, minimising the potential of disruptions and barriers that may occur when ingesting a request from Terra API to your server.

Authentication for AWS destinations

All AWS-based require the following authentication setup.

---

IAM User Access Key

The most basic way to allow Terra to write to your AWS resource is to create an IAM user with access limited to the resource you're trying to grant Terra access to. Attach relevant policies for access to the specific resource, (write access is generally the only one needed, unless specified otherwise)

---

Role-based access

In order to use role-based access, attach the following policy to your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::760292141147:role/EC2_Instance_Perms"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}

Data Structure

Data sent to SQS can either take the type of healthcheck or s3_payload. See for further details.

Each of these payloads will be a simple JSON payload formatted as in the diagram above. the url field in the data payload will be a download link from which you will need to download the data payload using a GET request. This is done to minimise the size of messages in your queue.

The URL will be a pre-signed link to an object stored in one of Terra's S3 buckets.

Note: Terra offers the possibility of using your own S3 bucket in combination with the SQS destination. For setting this up, kindly contact Terra support.