How to integrate Birdie + S3/Azure/GCS/

With the S3 connector, Birdie can import data from multiple file formats from AWS S3 or a storage service that implements the S3 API such as Google Cloud Storage. Once a day the connector checks if there are new objects (files) and if so imports the records in those objects.

• Dedicated bucket for Birdie Integration

• Birdie integration requires a service account with read-only access. Write access may be granted as optional to support teams during initial/manual dataset uploads.

• See docs on creating a user and generating an access key for the user.

• If you do not wish to provide a service account, see the docs to create a role and create an IAM Policy that allows Birdie (reach out and we'll provide the ID for the Birdie Account) to assume the role (see the docs)

• Create an IAM Policy that gives the user/role S3 Read or S3 Read/Write Access to the specific bucket. See the docs on how to this.

To enable secure access to an Azure Blob Storage container, you can use either Shared Access Signature (SAS) or Shared Key Authentication.

SAS Tokens are the recommended approach because they provide limited, temporary access without exposing the account key.

1. Shared Access Signature

   • To generate a SAS token for an Azure Blob Storage container using the Azure Portal, go to your storage account, select the container, and click on “Shared Access Signature.” Configure the token by selecting the permissions and setting the start and expiry time. Once configured, click “Generate SAS” and copy the SAS URL or token.
   • For more information about Shared Access Signature, see the docs
     
     

2. Shared Key Authentication

   • To access an Azure Blob Storage container using Shared Key Authentication, you use the storage account name and account key, which provide permanent, full access to the storage account. By supplying the account name, account key, and container name to your application or client, you can perform any operation on the blobs and containers.
   • For more information about Shared Key Authentication, see the docs

Note: Shared Key Authentication provides full, permanent access to your Azure storage account. Anyone with the account key can read, modify, or delete all containers and blobs. For this reason, do not share your account key with untrusted parties, and use it only for internal systems or administrative tasks. Consider using SAS tokens for temporary, limited access whenever possible to reduce security risks.

To enable HMAC Access for a Google Cloud Storage (GCS) bucket, please follow the instructions provided in the Google Cloud documentation.

Recommended Steps:

1. Create a New Service Account

   • Create a new Service Account with the necessary permissions to list buckets and read objects. Assign the "Storage Object Viewer" role to this Service Account.

2. Create a New HMAC Key

   • Follow the instructions to create a new HMAC Key for the newly created Service Account. This key will allow secure access to your GCS bucket.

3. Set Up IAM Policy

   • Bind an IAM policy that grants permissions to list objects only within the target bucket and, if necessary, a specific folder within that bucket. This ensures that access is restricted to the appropriate resources, enhancing security.

• For more information on how this works, Read about the GCS XML API, which works with S3 compatible tools.

• For more information about HMAC keys, Read about HMAC keys for GCS

• IAM Policy example with read-write access:

• Region: The region, e.g "us-west-2" (AWS) or "us-central1" (GCP).

• Bucket: The bucket name.

• Prefix: A prefix for the object keys. We suggest organizing it based on the kind of data (e.g birdie/tickets, birdie/nps)

• Format: The data format to use. Currently only supports parquet and csv.

• Kind: The kind of data you're trying to import. This defines what schema Birdie expects when reading rows from your file. Supported values are:

  • review

  • nps

  • csat

  • support_ticket

  • social_media_post

  • issue

  • accounts

• Credentials for S3

  • Access Key ID / HMAC Access ID

  • Secret Key ID / HMAC Secret

  • External ID (optional,AWS Specific)

  • Role ARN (optional, AWS specific)

  • The S3 endpoint to use. Only needed if not using AWS S3.

• Start Date: Date to filter objects by (object modified at).

Each row of the file must fit within one of the following schemas. The schema must match the kind selected when configuring the parameters.

See the oficial PARQUET spec for more information on the supported types and logical types.

Note 1: To ensure consistency, please upload only one row per conversation containing the survey response fields (such as survey_type, survey_title, rating, etc.). This message should be the final one for that ticket, reflecting the client’s closing thoughts on the service provided.

Note 2: To upload multiple messages per ticket, make sure the "ticket" fields (such as subject, status, priority, channel and tags) are consistent across all messages

Any columns that don't fit under the previously listed schemas may become custom fields.

The name of the column in the Parquet Schema must be configured as the key/source of the custom field inside the Birdie App.