> For the complete documentation index, see [llms.txt](https://docs.tryterra.co/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tryterra.co/destinations/queuing-services-sqs-kafka.md).

# Queues (SQS, Kafka)

Deliver Terra events to an AWS SQS queue or a Kafka topic, with each message linking to the full payload.

Terra can deliver events straight into an AWS SQS queue or publish them to a Kafka topic. Each message is small and carries a pre-signed link to the full data payload. This page covers setting up each service, adding the destination, and the message format.

## Before you begin

* An [AWS SQS](https://aws.amazon.com/sqs/) queue, or an [Apache Kafka](https://kafka.apache.org/) cluster with a topic for Terra to publish to.

## Set up AWS SQS

AWS SQS is a managed queuing system that lets Terra deliver events straight into your existing queue, minimising disruptions when ingesting data into your server.

Give Terra a role in your AWS account, or an access key. For a role, see [Role-based access](#role-based-access).

For an access key, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and attach a policy granting:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage"
      ],
      "Resource": "arn:aws:sqs:your-region:your-account-id:your-queue-name"
    }
  ]
}
```

If the queue is encrypted with a customer-managed KMS key, also allow `kms:GenerateDataKey` and `kms:Decrypt` on that key. If Terra uploads payloads to your own S3 bucket, the user also needs `s3:PutObject` on that bucket.

## Set up Kafka

Terra connects to Kafka using **SASL PLAIN** authentication over **TLS**. Create SASL credentials for Terra on your cluster.

Terra auto-creates the topic if it doesn't exist. This requires `auto.create.topics.enable=true` on the broker. If it is disabled, create the topic manually first.

## Add the destination in Terra

On the [Connections page](https://dashboard.tryterra.co/dashboard/connections) in your Terra dashboard, click **Add New**, add the queue destination, and enter the details for your service:

| Service | Details                                                                                    |
| ------- | ------------------------------------------------------------------------------------------ |
| AWS SQS | **Queue URL**, **Access Key ID**, **Secret Access Key**, and **region**                    |
| Kafka   | **Broker host and port**, **username and password** (SASL credentials), and **topic name** |

### Terra checks SQS access when you save

When you create an SQS destination, turn one on, or change its configuration, Terra first sends one test message to the queue. It uses the same credentials as deliveries, whether that is a role or an access key. If Terra uploads the destination's payloads to your own S3 bucket, it writes a small file, `terra-connection-check.json`, to the root of that bucket before sending the message. Every check overwrites that same key.

The message and the file have the same body:

```json
{
  "type": "connection_check",
  "message": "Terra sent this to check it can deliver here. You can ignore or delete it.",
  "checked_at": "2026-11-10T09:30:00Z"
}
```

If your consumer handles every message on the queue, have it delete `connection_check` messages without processing them. They carry no `url`.

If the check fails, Terra doesn't save the change and names the setting to fix. Through the API, that is a `400` with an entry such as `config.queue: this queue doesn't exist in this region`. If Terra can't reach AWS, the request returns a `503` and you can retry it. A destination saved turned off isn't checked, and a `?dry_run=true` request sends nothing.

## Role-based access

Instead of an access key, you can let Terra assume a role in your AWS account. Terra assumes it through its own AWS identity, so there is no key to store or rotate. You create the role from Terra's CloudFormation template. It can only send messages to the one queue you name, and write objects to your own S3 bucket if Terra uploads payloads there. It can't read, list or delete anything.

{% hint style="info" %}
The dashboard will offer this option soon. Until then, set it up through the [admin API](https://docs.tryterra.co/reference/admin-api/overview). The examples below use [`terra admin-api`](https://docs.tryterra.co/developer-tools/terra-cli/raw-requests) from the Terra CLI.
{% endhint %}

### Get your setup values

`GET /environments/{dev_id}/unified-api/destination-types` returns `aws_role_setup` on the `sqs` type. It needs the `destinations:read` scope.

```bash
terra admin-api /environments/<DEV_ID>/unified-api/destination-types --jq '.data[] | select(.type == "sqs") | .aws_role_setup'
```

| Field                 | What it is                                                                                                                 |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `external_id`         | The External ID Terra assumes your role with. It is unique to your Terra account and the same for every environment in it. |
| `terra_principal_arn` | Terra's AWS principal, the only identity your role trusts.                                                                 |
| `template_url`        | Terra's CloudFormation template.                                                                                           |
| `quick_create_url`    | Opens the AWS console's **Quick create stack** page with the template, `ExternalId` and `TerraPrincipalArn` filled in.     |

If `aws_role_setup` is missing, role-based access isn't available for that environment. Use an access key instead.

### Create the role

1. Sign in to the AWS account that owns the queue, and open `quick_create_url`.
2. Set `QueueArn` to your queue's ARN, `arn:aws:sqs:<REGION>:<ACCOUNT_ID>:<QUEUE_NAME>`.
3. If Terra uploads payloads to your own S3 bucket, set `BucketName` to that bucket. Otherwise leave it empty.
4. If the queue or bucket is encrypted with a customer-managed KMS key, set `KmsKeyArn` to the key's ARN. Otherwise leave it empty.
5. Leave `ExternalId` and `TerraPrincipalArn` as they are. Terra only assumes the role with your account's External ID, so a stack with a different value won't work.
6. Acknowledge that the stack creates an IAM role, and create the stack.
7. When the stack is created, copy `RoleArn` from its **Outputs** tab. Paste it rather than typing it: the role's name ends in the UUID from the stack's ID, as in `arn:aws:iam::<ACCOUNT_ID>:role/TerraByoCloud-<STACK_UUID>`.

Terra only accepts roles named `TerraByoCloud-<name>`, which is what the template creates. To deploy the template with your own tooling, use `template_url` with the same parameters.

### Add the role to the destination

Put the `RoleArn` in the destination's `config` as `role_arn`, when you create the destination or with a `PATCH`. Both need the `destinations:write` scope.

```bash
terra admin-api /environments/<DEV_ID>/unified-api/destinations -X POST --body '{
  "type": "sqs",
  "config": {
    "queue": "https://sqs.<REGION>.amazonaws.com/<ACCOUNT_ID>/<QUEUE_NAME>",
    "region": "<REGION>",
    "role_arn": "<ROLE_ARN>"
  }
}'
```

A destination uses a role or an access key, not both. To move an existing destination from an access key to the role, clear the key in the same request:

```bash
terra admin-api /environments/<DEV_ID>/unified-api/destinations/<DESTINATION_ID> -X PATCH --body '{
  "config": {
    "role_arn": "<ROLE_ARN>",
    "access_key_id": null,
    "secret_access_key": null
  }
}'
```

To go back to an access key, send both keys with `"role_arn": null`. Saving runs the [access check](#terra-checks-sqs-access-when-you-save), so a role Terra can't assume or send with is refused before any delivery uses it.

## Message format

SQS and Kafka messages use the same format. Each message is a JSON payload containing a download URL:

```json
{
  "status": "success",
  "type": "s3_payload",
  "url": "https://storage.googleapis.com/terra-services-p-bkt-payloads/...",
  "expires_in": 600,
  "payload_reference": "bb0a8574-ac33-42e5-ae33-49aa52600b1e"
}
```

* `url`: a pre-signed link to the full data payload, hosted on Terra's infrastructure. Download it with a `GET` request; no credentials are needed.
* `expires_in`: URL validity in seconds (600 = 10 minutes).
* `payload_reference`: a unique identifier for this payload delivery.

This approach keeps message sizes small. The actual data payload must be fetched from the URL.

On Kafka, messages are published without a partition key (distributed across partitions by least-bytes balancing).

### Payloads in your own S3 bucket

For SQS, Terra can upload payloads to **your own S3 bucket** instead of its own storage. Contact Terra support to set this up.

The message format stays the same, but `url` is then the object's plain S3 address, `https://<bucket>.s3.<region>.amazonaws.com/<key>`. It is not pre-signed:

* Read it with your own AWS credentials, for example with the AWS SDK's `GetObject` on that bucket and key. A plain `GET` without credentials is refused.
* `expires_in` does not apply. The object stays in your bucket until you delete it or a lifecycle rule removes it.
* With role-based access, the role must be able to write to that bucket: set `BucketName` when you [create the role](#create-the-role).

## Next steps

* [User authentication](/unified-api/authentication-flow.md): link a user's wearable so events start flowing.
* [Event delivery](/destinations/events.md): see what each delivery status means and how retries work.
* [Health data events](/unified-api/receiving-data-updates.md): process the data Terra delivers.
