Queuing services (SQS, Kafka)

AWS SQS (Simple Queue Service)

AWS SQS is a managed queuing system that lets Terra deliver events straight into your existing queue, minimising disruptions when ingesting data into your server.

Authentication

Create an IAM user with permissions to send messages to your queue. Attach a policy granting at minimum:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:GetQueueUrl"
      ],
      "Resource": "arn:aws:sqs:your-region:your-account-id:your-queue-name"
    }
  ]
}

Enter the queue URL, Access Key ID, Secret Access Key, and region in the Terra Dashboard when adding the SQS destination.

Message format

Each SQS message is a JSON payload containing a pre-signed download URL:

{
  "status": "success",
  "type": "s3_payload",
  "url": "https://storage.googleapis.com/terra-services-p-bkt-payloads/...",
  "expires_in": 600,
  "payload_reference": "bb0a8574-ac33-42e5-ae33-49aa52600b1e"
}

• url — A pre-signed link to the full data payload, hosted on Terra's infrastructure. Download it with a GET request.

• expires_in — URL validity in seconds (600 = 10 minutes).

• payload_reference — A unique identifier for this payload delivery.

This approach keeps message sizes small. The actual data payload must be fetched from the URL.

Terra can optionally upload payloads to your own S3 bucket instead of Terra's storage, so the URL points to your bucket. Contact Terra support to set this up.

---

Kafka

Apache Kafka lets Terra publish events to a topic in your Kafka cluster.

Authentication

Terra connects to Kafka using SASL PLAIN authentication over TLS. You'll need to provide:

• Broker host and port

• Username and password (SASL credentials)

• Topic name — Terra will auto-create the topic if it doesn't exist (requires auto.create.topics.enable=true on the broker; if disabled, create the topic manually first)

Enter these details in the Terra Dashboard when adding the Kafka destination.

Message format

Kafka messages use the same format as SQS:

{
  "status": "success",
  "type": "s3_payload",
  "url": "https://storage.googleapis.com/terra-services-p-bkt-payloads/...",
  "expires_in": 600,
  "payload_reference": "bb0a8574-ac33-42e5-ae33-49aa52600b1e"
}

Messages are published without a partition key (distributed across partitions by least-bytes balancing). The url field contains a pre-signed link (10 minute expiry) to the full data payload. As with SQS, payload_reference is a unique identifier for this payload delivery.