# Queuing services (SQS, Kafka)

## AWS SQS (Simple Queue Service)

[AWS SQS](https://aws.amazon.com/sqs/) is a managed queuing system that lets Terra deliver events straight into your existing queue, minimising disruptions when ingesting data into your server.

### Authentication

[Create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) with permissions to send messages to your queue. Attach a policy granting at minimum:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:your-region:your-account-id:your-queue-name"
    }
  ]
}
```

Enter the **queue URL**, **Access Key ID**, **Secret Access Key**, and **region** in the Terra Dashboard when adding the SQS destination.

### Message format

Each SQS message is a JSON payload containing a pre-signed download URL:

```json
{
  "status": "success",
  "type": "s3_payload",
  "url": "https://presigned-download-url.example.com/...",
  "expires_in": 600
}
```

* `url` — A pre-signed link to the full data payload, hosted on Terra's infrastructure. Download it with a `GET` request.
* `expires_in` — URL validity in seconds (600 = 10 minutes).

This approach keeps message sizes small. The actual data payload must be fetched from the URL.

{% hint style="info" %}
Terra can optionally upload payloads to **your own S3 bucket** instead of Terra's storage, so the URL points to your bucket. Contact Terra support to set this up.
{% endhint %}

***

## Kafka

[Apache Kafka](https://kafka.apache.org/) lets Terra publish events to a topic in your Kafka cluster.

### Authentication

Terra connects to Kafka using **SASL PLAIN** authentication over **TLS**. You'll need to provide:

* **Broker host and port**
* **Username and password** (SASL credentials)
* **Topic name** — Terra will auto-create the topic if it doesn't exist (requires `auto.create.topics.enable=true` on the broker; if disabled, create the topic manually first)

Enter these details in the Terra Dashboard when adding the Kafka destination.

### Message format

Kafka messages use the same format as SQS:

```json
{
  "status": "success",
  "type": "s3_payload",
  "url": "https://presigned-download-url.example.com/...",
  "expires_in": 600
}
```

Messages are published without a partition key (distributed across partitions by least-bytes balancing). The `url` field contains a pre-signed link (10 minute expiry) to the full data payload.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.tryterra.co/health-and-fitness-api/integration-setup/setting-up-data-destinations/queuing-services-sqs-kafka.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.