Databricks

This article describes how to configure your Databricks workspace to allow Birdie to export data to a dedicated catalog and schema.

Prerequisite: Before configuring the export integration, you should already have completed the setup described in Data Ingestion with Databricks. The steps below assume a Birdie service principal (or user) already exists and can use a SQL Warehouse.

---

Overview

Birdie can load processed data (e.g., enriched feedback, areas, opportunities) into your Databricks workspace. We recommend creating:

• A separate schema for Birdie exports, so write access stays isolated from the tables Birdie reads

• A Unity Catalog volume in that schema, used to stage the files before they are loaded into tables

Step 1 – Create a schema for Birdie exports

CREATE SCHEMA IF NOT EXISTS main.birdie_export;

Replace main with the catalog you want the export to live in. Birdie manages every table in this schema, so it should not be shared with tables of your own.

Step 2 – Create a volume to stage the files

Birdie uploads each export file to a volume before loading it into a table. Unity Catalog volumes are the supported staging area for PUT uploads through a SQL Warehouse.

CREATE VOLUME IF NOT EXISTS main.birdie_export.birdie_stage;

Files are removed from the volume as soon as they are loaded, so it holds no data between exports.

Step 3 – Grant the required privileges

Grant the Birdie principal the privileges it needs on the catalog, the export schema and the staging volume. If you created a service principal or user during the data ingestion setup, grant these on that same principal — Birdie can use one identity for both reading and writing.

-- Catalog access
GRANT USE CATALOG ON CATALOG main TO `birdie`;

-- The export schema: Birdie creates, replaces and writes its own tables here
GRANT USE SCHEMA ON SCHEMA main.birdie_export TO `birdie`;
GRANT CREATE TABLE ON SCHEMA main.birdie_export TO `birdie`;
GRANT SELECT ON SCHEMA main.birdie_export TO `birdie`;
GRANT MODIFY ON SCHEMA main.birdie_export TO `birdie`;

-- The staging volume
GRANT READ VOLUME ON VOLUME main.birdie_export.birdie_stage TO `birdie`;
GRANT WRITE VOLUME ON VOLUME main.birdie_export.birdie_stage TO `birdie`;

Replace `birdie` with your service principal's application ID (OAuth) or the user account that owns the token (PAT).

The principal also needs CAN USE on the SQL Warehouse that runs the load. If this was already granted during the ingestion setup, the same warehouse can be reused.

Do not create the destination tables yourself. Unity Catalog gives a table's creator rights over it that these grants do not need to duplicate, and Birdie relies on that: every table this export manages has to be one Birdie's own identity created, or a later load can fail on ownership even with all of the grants above in place.

Step 4 – Share the export configuration with Birdie

Share the following details securely with Birdie:

• Catalog name (e.g. main)

• Export schema name (e.g. birdie_export)

• Staging volume name (e.g. birdie_stage) — required; the export cannot stage files without it

If not already shared during the data ingestion setup, also include:

• Workspace host (e.g. https://dbc-xxxx.cloud.databricks.com)

• SQL Warehouse ID

• Service principal client ID and client secret, or the personal access token

Share credentials securely with the Birdie team. Never send credentials via email or unencrypted channels.

Birdie will configure the export connector and confirm once the integration is active.

---

How the Export Works

1. Staging upload: each file is split into parts of at most 100,000 rows, converted to Parquet, and uploaded into a dedicated folder in the staging volume. Every run writes to its own folder, so a part left behind by an interrupted run is never loaded twice.

2. Staging table: the parts are loaded via COPY INTO into a temporary table, with every column as a string.

3. Table creation: on the first export of a file, the destination table is created from the columns the file carries.

4. Load: for incremental files, the rows whose primary keys the file carries are deleted and re-inserted, and every other row of the table stays untouched. Full files are replaced whole in a single statement.

5. Cleanup: the staged files are removed from the volume and the staging table is dropped.

For incremental files, the delete and the insert run as two separate statements rather than one atomic merge, unlike Snowflake's export. A table read in between the two briefly shows that file's rows as missing.

Exported Data Format

Birdie creates one table per exported file. Table and column naming follows our standard CSV export conventions, so column names such as Feedback ID keep the spaces.

The tables are created with Delta column mapping enabled, since Delta otherwise rejects column names containing spaces. If you read these tables from outside Databricks, your Delta reader needs to support column mapping.

Every column is loaded as a string, except:

• Rating and Total Messages, loaded as numeric

• Posted At, Ingested At and Updated At, loaded as timestamps

A value that cannot be read as that type is loaded as empty rather than failing the export.

Each table also carries a Birdie Exported At column holding the time of the run that wrote the row.

Data is de-duplicated on each load, so it will be ready to use. For details on joining tables, refer to our data exports documentation.

Some data (e.g., custom fields) may require additional handling due to their object-like nature.

References

• Unity Catalog Privileges

• Unity Catalog Volumes

• COPY INTO

• Data Ingestion with Databricks

• Data exports