Cloud storage (S3, GCP)

Terra allows you to connect an S3 Bucket as a destination, to get all data dumped directly into a bucket of your choice.

Authentication

AWS

All AWS-based destinations follow the same authentication setup.

IAM User Access Key

The most basic way to allow Terra to write to your AWS resource is to create an IAM user with access limited to the resource you're trying to grant Terra access to. Attach relevant policies for access to the specific resource, (write access is generally the only one needed, unless specified otherwise)

Role-based access

In order to use role-based access, attach the following policy to your bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::760292141147:role/EC2_Instance_Perms"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}

GCP

For GCP destinations, you'll need to create a service account, and generate credentials for it. See the guide here for further details. once you have generated credentials, download the JSON file with the credentials you just created, and upload it in the corresponding section when setting up your GCP-based destination

---

Data Structure

When data is sent to your S3 Bucket or GCS, it will be dumped using the following folder structure

Example for AWS S3

Versioned objects will be placed under the appropriate API version (in the screenshot above, this corresponds to 2022-03-16. Non versioned objects (e.g. authentication Events) will be placed in their appropriate event type folder, outside of the version folder

In all, every event will have as a parent folder the Event Type which it corresponds to, and will be saved with a unique name identifying it.

As shown above, the name will either be a concatenation of one of the below:

• For Data Events: the user ID & the start time of the period the event refers to

• For all other Event Types: the user ID & timestamp the event was generated at