> For the complete documentation index, see [llms.txt](https://docs.tryterra.co/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.tryterra.co/unified-api/integration-setup/setting-up-data-destinations/queuing-services-sqs-kafka.md).

# Queuing services (SQS, Kafka)

## AWS SQS (Simple Queue Service)

[AWS SQS](https://aws.amazon.com/sqs/) is a managed queuing system that lets Terra deliver events straight into your existing queue, minimising disruptions when ingesting data into your server.

### Authentication

[Create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) with permissions to send messages to your queue. Attach a policy granting at minimum:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:your-region:your-account-id:your-queue-name"
    }
  ]
}
```

Enter the **queue URL**, **Access Key ID**, **Secret Access Key**, and **region** in the Terra Dashboard when adding the SQS destination.

### Message format

Each SQS message is a JSON payload containing a pre-signed download URL:

```json
{
  "status": "success",
  "type": "s3_payload",
  "url": "https://presigned-download-url.example.com/...",
  "expires_in": 600,
  "user_id": "0d09d3a4-f6e8-4f1c-9a3e-2b6c1d5e7f80",
  "reference_id": "your-user-ref-123"
}
```

* `url` — A pre-signed link to the full data payload, hosted on Terra's infrastructure. Download it with a `GET` request.
* `expires_in` — URL validity in seconds (600 = 10 minutes).
* `user_id` — The Terra user ID the payload belongs to.
* `reference_id` — The `reference_id` you set when connecting the user (empty if none was set).

This approach keeps message sizes small. The actual data payload must be fetched from the URL. `user_id` and `reference_id` are included on the message itself so you can identify and re-request a payload even if its `url` expires before your consumer fetches it.

{% hint style="info" %}
Terra can optionally upload payloads to **your own S3 bucket** instead of Terra's storage, so the URL points to your bucket. Contact Terra support to set this up.
{% endhint %}

***

## Kafka

[Apache Kafka](https://kafka.apache.org/) lets Terra publish events to a topic in your Kafka cluster.

### Authentication

Terra connects to Kafka using **SASL PLAIN** authentication over **TLS**. You'll need to provide:

* **Broker host and port**
* **Username and password** (SASL credentials)
* **Topic name** — Terra will auto-create the topic if it doesn't exist (requires `auto.create.topics.enable=true` on the broker; if disabled, create the topic manually first)

Enter these details in the Terra Dashboard when adding the Kafka destination.

### Message format

Kafka messages use the same format as SQS:

```json
{
  "status": "success",
  "type": "s3_payload",
  "url": "https://presigned-download-url.example.com/...",
  "expires_in": 600,
  "user_id": "0d09d3a4-f6e8-4f1c-9a3e-2b6c1d5e7f80",
  "reference_id": "your-user-ref-123"
}
```

Messages are published without a partition key (distributed across partitions by least-bytes balancing). The `url` field contains a pre-signed link (10 minute expiry) to the full data payload. As with SQS, `user_id` and `reference_id` identify the user the payload belongs to, so you can re-request it if the `url` expires before you fetch it.