S3 / Azure / GCS

Overview

With the S3 connector, Birdie can import data from multiple file formats from AWS S3 or a storage service that implements the S3 API such as Google Cloud Storage. Once a day the connector checks if there are new objects (files) and if so imports the records in those objects.

Requirements

• Dedicated bucket (or prefix) for Birdie ingestion.

• Service account/role with read access to the bucket and its objects (s3:GetObject, s3:ListBucket, or the equivalent for Azure/GCS). For KMS-encrypted buckets, also grant kms:Decrypt on the encryption key.

Write access is optional. Granting it (scoped to the birdie/_partitioned/* prefix) lets Birdie split large source files in place — see Handling large source files. Without it, ingestion runs read-only and Birdie splits the files in a temporary Birdie-owned environment instead.

Setup in S3-compatible storage

AWS

• See docs on creating a user and generating an access key for the user.

• If you do not wish to provide a service account, see the docs to create a role and create an IAM Policy that allows Birdie (reach out and we'll provide the ID for the Birdie Account) to assume the role (see the docs).

• Create an IAM Policy that gives the user/role S3 Read or S3 Read/Write Access to the specific bucket. See the docs on how to this. Example with read on the bucket and write scoped to the partitioning prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BirdieRead",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::YOUR_BUCKET",
        "arn:aws:s3:::YOUR_BUCKET/*"
      ]
    },
    {
      "Sid": "BirdiePartitionInPlace",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET/birdie/_partitioned/*"
    }
  ]
}

Birdie's access is read-only by default. The BirdiePartitionInPlace statement is the only write access requested, and it is optional. It is scoped to the birdie/_partitioned/* prefix and used only to split large source files into smaller chunks inside your own bucket, so your data never leaves your storage account (see Handling large source files). Omit that statement to keep the role read-only; Birdie then splits the files in a temporary Birdie-owned environment instead.

Azure

To enable secure access to an Azure Blob Storage container, the recommended approach is to use Azure AD authentication with a Service Principal.

This method provides secure, role-based access control (RBAC) without exposing storage account keys.

As a fallback, Shared Key Authentication can be used when Azure AD is not available.

Why Service Principal is Recommended

Using a Service Principal allows secure, scoped access to a specific Storage Account via RBAC.

Instead of sharing account keys, you grant explicit permissions to an application identity.

No storage account keys are exposed, only role-based access control (RBAC) is granted, the credentials are revocable, secrets can be rotated.

This is the enterprise security best practice.

Service Principal (Azure AD)

Step 1. Create a Storage Account

• Go to Azure Portal;

• Navigate to Storage accounts;

• Click Create;

• Provide:

  • Subscription;

  • Resource Group;

  • Storage account name (lowercase, globally unique);

  • Region.

• Click Review + Create.

The Blob endpoint will be:

https://<storage-account-name>.blob.core.windows.net

Step 2 — Create a Container

• Open your Storage Account;

• Go to Containers;

• Click + Container;

• Define the container name.

This will be used as the bucket.

Step 3 — Create a Service Principal

• Navigate to App registrations;

• Click New registration;

• Provide a name;

• Click Register.

• Copy Application (client) ID;

• Copy Directory (tenant) ID.

Step 4 — Generate a Client Secret

• Inside the App Registration;

• Go to Certificates & secrets;

• Click New client secret;

• Set expiration;

• Copy the Secret VALUE (not the Secret ID).

Step 5 — Assign Storage Permissions

• Go to your Storage Account;

• Open Access Control (IAM);

• Click Add role assignment;

• Select role: Storage Blob Data Contributor;

• Assign access to:User, group, or service principal;

• Select your App Registration;

• Save.

To configure the integration, you'll need to share with Birdie: Directory (tenant) ID, Application (client) ID, Client Secret VALUE, Endpoint, Region and Container name.

Shared Key Authentication (fallback)

Shared Key Authentication provides full, permanent access to the entire storage account.

Anyone with the key can:

• Read all blobs

• Modify data

• Delete containers

Use this method only when Azure AD authentication is not possible.

In order to retrieve the Shared Key:

• Go to Storage Account

• Navigate to Access keys

• Copy:

  • Storage account name

  • Key1 or Key2

To configure the integration, you'll need to share with Birdie: Storage account name, Access key, Endpoint and Container name.

GCP

To enable HMAC Access for a Google Cloud Storage (GCS) bucket, please follow the instructions provided in the Google Cloud documentation.

Recommended Steps:

1. Create a New Service Account

   • Create a new Service Account with the necessary permissions to list buckets and read objects. Assign the "Storage Object Viewer" role to this Service Account.

2. Create a New HMAC Key

   • Follow the instructions to create a new HMAC Key for the newly created Service Account. This key will allow secure access to your GCS bucket.

3. Set Up IAM Policy

   • Bind an IAM policy that grants permissions to list objects only within the target bucket and, if necessary, a specific folder within that bucket. This ensures that access is restricted to the appropriate resources, enhancing security.

• For more information on how this works, Read about the GCS XML API, which works with S3 compatible tools.

• For more information about HMAC keys, Read about HMAC keys for GCS.

Handling large source files

Birdie lists files by modification time and processes them in time-bounded import windows (typically per day). When a single source parquet file is too large to fit one window — empirically, above ~500,000 rows — Birdie splits it into 500k-row chunks before parsing. Where those chunks are written depends on the access granted on the source bucket:

• With write access (recommended): Birdie writes the chunks back to the same bucket under birdie/_partitioned/{org_id}/{kind}/.... Your data stays inside your storage account, with no cross-cloud transfer.

• Read-only access: Birdie splits the chunks in a temporary Birdie-owned working environment, ingests from there, and removes the intermediate copies once ingestion finishes. The result is the same, but it adds a cross-cloud transfer and Birdie holds a short-lived copy of your data while ingestion runs. Reach out if you would rather keep everything inside your own account.

If your source files are already small enough to fit one window (for example one file per hour, under ~500k rows each), no splitting is needed and read-only access is enough.

Connect to Birdie

To configure the connector, provide Birdie with the following parameters and credentials.

Parameters

• Region: The region, e.g "us-west-2" (AWS) or "us-central1" (GCP).

• Bucket: The bucket name. For audio ingestion, this is also where the audio recordings must live.

• Prefix: A prefix for the object keys. We suggest organizing it based on the kind of data (e.g birdie/tickets, birdie/nps).

• Format: The data format to use. Currently only supports parquet and csv.

• Kind: The kind of data you're trying to import. This defines what schema Birdie expects when reading rows from your file. Supported values are:

  • review

  • nps

  • csat

  • support_ticket

  • complaint

  • social_media_post

  • issue

  • account

  • survey

  • audio_support_ticket

  • audio_complaint

• Credentials for S3

  • Access Key ID / HMAC Access ID

  • Secret Key ID / HMAC Secret

  • External ID (optional, AWS specific)

  • Role ARN (optional, AWS specific)

  • The S3 endpoint to use. Only needed if not using AWS S3.

• Start Date: Date to filter objects by (object modified at).

Data in scope

S3 Schemas

Each row of the file must fit within one of the following schemas. The schema must match the kind selected when configuring the parameters.

See the oficial PARQUET spec for more information on the supported types and logical types.

Row-level schema definitions (columns, types, required fields) are maintained centrally in the data ingestion schema documentation.

Feedback — a single-point interaction (one person, one moment, one piece of input):

• Feedbacks - Review

• Feedbacks - NPS and CSAT

Conversation — a multi-turn interaction, denormalized into one row per message:

• Conversations - Support tickets

• Conversations - Complaints

• Conversations - Social Media Post

• Conversations - Issue

Account — defines the "Who" behind the data, linking Feedback and Conversation records to a customer profile:

• Accounts

Audio

Audio recordings must live in the same bucket as the base file, and are read the same way as any other kind: one row per call in the base file, with a column pointing at the corresponding recording.

Audio is currently only supported for the following kinds:

• Support Ticket

• Complaint

Supported recording formats:

• WAV

• MP3

See Audio // Conversation // Support tickets and Audio // Conversation // Complaints in the data ingestion schema documentation for the row schema of each kind.

Custom Fields

Any columns that don't fit under the previously listed schemas may become custom fields.

The name of the column in the Parquet Schema must be configured as the key/source of the custom field inside the Birdie App.