> For the complete documentation index, see [llms.txt](https://ask.birdie.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ask.birdie.ai/integrations-and-data-ingestion/how-to-integrate-with/s3-azure-gcs.md). # S3 / Azure / GCS ### Overview With the S3 connector, Birdie can import data from multiple file formats from AWS S3 or a storage service that implements the S3 API such as Google Cloud Storage. Once a day the connector checks if there are new objects (files) and if so imports the records in those objects. ### Requirements * Dedicated bucket (or prefix) for Birdie ingestion. * Service account/role with **read** access to the bucket and its objects (`s3:GetObject`, `s3:ListBucket`, or the equivalent for Azure/GCS). For KMS-encrypted buckets, also grant `kms:Decrypt` on the encryption key. {% hint style="info" %} Write access is optional. Granting it (scoped to the `birdie/_partitioned/*` prefix) lets Birdie split large source files in place — see [Handling large source files](#handling-large-source-files). Without it, ingestion runs read-only and Birdie splits the files in a temporary Birdie-owned environment instead. {% endhint %} ### Setup in S3-compatible storage #### AWS * See docs on creating a [user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) and generating an [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for the user. * If you do not wish to provide a service account, see the docs to [create a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html#roles-creatingrole-user-console) and create an IAM Policy that allows Birdie (reach out and we'll provide the ID for the Birdie Account) to assume the role (see the [docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html#tutorial_cross-account-with-roles-2)). * Create an IAM Policy that gives the user/role S3 Read or S3 Read/Write Access to the specific bucket. See the [docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket-console.html) on how to this. Example with read on the bucket and write scoped to the partitioning prefix: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "BirdieRead", "Effect": "Allow", "Action": ["s3:GetObject", "s3:ListBucket"], "Resource": [ "arn:aws:s3:::YOUR_BUCKET", "arn:aws:s3:::YOUR_BUCKET/*" ] }, { "Sid": "BirdiePartitionInPlace", "Effect": "Allow", "Action": ["s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"], "Resource": "arn:aws:s3:::YOUR_BUCKET/birdie/_partitioned/*" } ] } ``` {% hint style="info" %} Birdie's access is read-only by default. The `BirdiePartitionInPlace` statement is the only write access requested, and it is optional. It is scoped to the `birdie/_partitioned/*` prefix and used only to split large source files into smaller chunks inside your own bucket, so your data never leaves your storage account (see [Handling large source files](#handling-large-source-files)). Omit that statement to keep the role read-only; Birdie then splits the files in a temporary Birdie-owned environment instead. {% endhint %} #### Azure To enable secure access to an Azure Blob Storage container, the recommended approach is to use **Azure AD authentication with a Service Principal**. This method provides secure, role-based access control (RBAC) without exposing storage account keys. As a fallback, **Shared Key Authentication** can be used when Azure AD is not available. {% hint style="info" %} **Why Service Principal is Recommended** Using a Service Principal allows secure, scoped access to a specific Storage Account via RBAC. Instead of sharing account keys, you grant explicit permissions to an application identity. No storage account keys are exposed, only role-based access control (RBAC) is granted, the credentials are revocable, secrets can be rotated. This is the enterprise security best practice. {% endhint %} **Service Principal (Azure AD)** **Step 1. Create a Storage Account** * Go to Azure Portal; * Navigate to [**Storage accounts**](https://portal.azure.com/#view/Microsoft_Azure_StorageHub/StorageHub.MenuView/~/StorageAccountsBrowse)**;** * Click **Create;** * Provide: * Subscription; * Resource Group; * Storage account name (lowercase, globally unique); * Region. * Click **Review + Create.** The Blob endpoint will be: ``` https://.blob.core.windows.net ``` **Step 2 — Create a Container** * Open your Storage Account; * Go to **Containers;** * Click **+ Container;** * Define the container name. This will be used as the `bucket`. **Step 3 — Create a Service Principal** * Navigate to [**App registrations**](https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationsListBlade)**;** * Click **New registration;** * Provide a name; * Click **Register.** * Copy **Application (client) ID;** * Copy **Directory (tenant) ID.** **Step 4 — Generate a Client Secret** * Inside the App Registration; * Go to **Certificates & secrets;** * Click **New client secret;** * Set expiration; * Copy the **Secret VALUE** (*not* the Secret ID). **Step 5 — Assign Storage Permissions** * Go to your **Storage Account;** * Open **Access Control (IAM);** * Click **Add role assignment;** * Select role: `Storage Blob Data Contributor`; * Assign access to:`User, group, or service principal`; * Select your App Registration; * Save. {% hint style="info" %} **To configure the integration, you'll need to share with Birdie:** Directory (tenant) ID, Application (client) ID, Client Secret VALUE, Endpoint, Region and Container name. {% endhint %} **Shared Key Authentication (fallback)** Shared Key Authentication provides full, **permanent** access to the entire storage account. Anyone with the key can: * Read all blobs * Modify data * Delete containers Use this method **only** when Azure AD authentication is not possible. In order to retrieve the Shared Key: * Go to **Storage Account** * Navigate to **Access keys** * Copy: * Storage account name * Key1 or Key2 {% hint style="info" %} **To configure the integration, you'll need to share with Birdie:** Storage account name, Access key, Endpoint and Container name. {% endhint %} #### GCP To enable HMAC Access for a Google Cloud Storage (GCS) bucket, please follow the instructions provided in the [Google Cloud documentation](https://cloud.google.com/storage/docs/authentication/managing-hmackeys?hl=pt-br#create). **Recommended Steps:** 1. Create a New Service Account * Create a new Service Account with the necessary permissions to list buckets and read objects. Assign the "Storage Object Viewer" role to this Service Account. 2. Create a New HMAC Key * Follow the [instructions](https://cloud.google.com/storage/docs/authentication/managing-hmackeys?hl=pt-br#create) to create a new HMAC Key for the newly created Service Account. This key will allow secure access to your GCS bucket. 3. Set Up IAM Policy * Bind an IAM policy that grants permissions to list objects only within the target bucket and, if necessary, a specific folder within that bucket. This ensures that access is restricted to the appropriate resources, enhancing security. * For more information on how this works, Read about the [GCS XML API](https://cloud.google.com/storage/docs/interoperability?hl=pt-br#xml_api), which works with S3 compatible tools. * For more information about HMAC keys, Read about [HMAC keys for GCS](https://cloud.google.com/storage/docs/authentication/hmackeys?hl=pt-br#overview). ### Handling large source files Birdie lists files by modification time and processes them in time-bounded import windows (typically per day). When a single source parquet file is too large to fit one window — empirically, above \~500,000 rows — Birdie splits it into 500k-row chunks before parsing. Where those chunks are written depends on the access granted on the source bucket: * **With write access** (recommended): Birdie writes the chunks back to the same bucket under `birdie/_partitioned/{org_id}/{kind}/...`. Your data stays inside your storage account, with no cross-cloud transfer. * **Read-only access**: Birdie splits the chunks in a temporary Birdie-owned working environment, ingests from there, and removes the intermediate copies once ingestion finishes. The result is the same, but it adds a cross-cloud transfer and Birdie holds a short-lived copy of your data while ingestion runs. Reach out if you would rather keep everything inside your own account. If your source files are already small enough to fit one window (for example one file per hour, under \~500k rows each), no splitting is needed and read-only access is enough. ### Connect to Birdie To configure the connector, provide Birdie with the following parameters and credentials. #### Parameters * Region: The region, e.g "us-west-2" (AWS) or "us-central1" (GCP). * Bucket: The bucket name. For audio ingestion, this is also where the audio recordings must live. * Prefix: A prefix for the object keys. We suggest organizing it based on the kind of data (e.g birdie/tickets, birdie/nps). * Format: The data format to use. Currently only supports parquet and csv. * Kind: The kind of data you're trying to import. This defines what schema Birdie expects when reading rows from your file. Supported values are: * review * nps * csat * support\_ticket * complaint * social\_media\_post * issue * account * survey * audio\_support\_ticket * audio\_complaint * Credentials for S3 * Access Key ID / HMAC Access ID * Secret Key ID / HMAC Secret * External ID (optional, AWS specific) * Role ARN (optional, AWS specific) * The S3 endpoint to use. Only needed if not using AWS S3. * Start Date: Date to filter objects by (object modified at). ### Data in scope #### S3 Schemas Each row of the file must fit within one of the following schemas. The schema must match the kind selected when configuring the parameters. See the oficial PARQUET spec for more information on the [supported types](https://github.com/apache/parquet-format?tab=readme-ov-file#types) and [logical types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md). Row-level schema definitions (columns, types, required fields) are maintained centrally in the [data ingestion schema documentation](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md). [**Feedback**](https://ask.birdie.ai/integrations-and-data-ingestion/how-to-integrate-with/pages/5Z8b5BkvkdE2oYhrsKO8#id-2.-understanding-record-types) — a single-point interaction (one person, one moment, one piece of input): * [Feedbacks - Review](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#feedbacks-review) * [Feedbacks - NPS and CSAT](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#feedbacks-nps-and-csat) [**Conversation**](https://ask.birdie.ai/integrations-and-data-ingestion/how-to-integrate-with/pages/5Z8b5BkvkdE2oYhrsKO8#id-2.-understanding-record-types) — a multi-turn interaction, denormalized into one row per message: * [Conversations - Support tickets](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#conversations-support-tickets) * [Conversations - Complaints](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#conversations-complaints) * [Conversations - Social Media Post](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#conversation-social-media-post) * [Conversations - Issue](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#conversation-issue) [**Account**](https://ask.birdie.ai/integrations-and-data-ingestion/how-to-integrate-with/pages/5Z8b5BkvkdE2oYhrsKO8#id-2.-understanding-record-types) — defines the "Who" behind the data, linking Feedback and Conversation records to a customer profile: * [Accounts](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#accounts) #### Audio Audio recordings must live in the same bucket as the base file, and are read the same way as any other kind: one row per call in the base file, with a column pointing at the corresponding recording. Audio is currently only supported for the following kinds: * Support Ticket * Complaint Supported recording formats: * WAV * MP3 See [Audio // Conversation // Support tickets](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#audio-conversation-support-tickets) and [Audio // Conversation // Complaints](/integrations-and-data-ingestion/data-schema-definitions-for-birdie-ingestion.md#audio-conversation-complaints) in the data ingestion schema documentation for the row schema of each kind. #### Custom Fields Any columns that don't fit under the previously listed schemas may become custom fields. The name of the column in the Parquet Schema must be configured as the key/source of the custom field inside the Birdie App. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://ask.birdie.ai/integrations-and-data-ingestion/how-to-integrate-with/s3-azure-gcs.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.