> For the complete documentation index, see [llms.txt](https://ask.birdie.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ask.birdie.ai/integrations-and-data-ingestion/how-to-export-with/databricks.md). # Databricks This article describes how to configure your Databricks workspace to allow Birdie to export data to a dedicated catalog and schema. > **Prerequisite:** Before configuring the export integration, you should already have completed the setup described in [Data Ingestion with Databricks](/integrations-and-data-ingestion/how-to-integrate-with/databricks.md). The steps below assume a Birdie service principal (or user) already exists and can use a SQL Warehouse. *** ### Overview Birdie can load processed data (e.g., enriched feedback, areas, opportunities) into your Databricks workspace. We recommend creating: * A separate **schema** for Birdie exports, so write access stays isolated from the tables Birdie reads * A **Unity Catalog volume** in that schema, used to stage the files before they are loaded into tables ### Step 1 – Create a schema for Birdie exports ```sql CREATE SCHEMA IF NOT EXISTS main.birdie_export; ``` Replace `main` with the catalog you want the export to live in. Birdie manages every table in this schema, so it should not be shared with tables of your own. ### Step 2 – Create a volume to stage the files Birdie uploads each export file to a volume before loading it into a table. Unity Catalog volumes are the supported staging area for `PUT` uploads through a SQL Warehouse. ```sql CREATE VOLUME IF NOT EXISTS main.birdie_export.birdie_stage; ``` Files are removed from the volume as soon as they are loaded, so it holds no data between exports. ### Step 3 – Grant the required privileges Grant the Birdie principal the privileges it needs on the catalog, the export schema and the staging volume. If you created a service principal or user during the data ingestion setup, grant these on that same principal — Birdie can use one identity for both reading and writing. ```sql -- Catalog access GRANT USE CATALOG ON CATALOG main TO `birdie`; -- The export schema: Birdie creates, replaces and writes its own tables here GRANT USE SCHEMA ON SCHEMA main.birdie_export TO `birdie`; GRANT CREATE TABLE ON SCHEMA main.birdie_export TO `birdie`; GRANT SELECT ON SCHEMA main.birdie_export TO `birdie`; GRANT MODIFY ON SCHEMA main.birdie_export TO `birdie`; -- The staging volume GRANT READ VOLUME ON VOLUME main.birdie_export.birdie_stage TO `birdie`; GRANT WRITE VOLUME ON VOLUME main.birdie_export.birdie_stage TO `birdie`; ``` Replace `` `birdie` `` with your service principal's application ID (OAuth) or the user account that owns the token (PAT). The principal also needs **CAN USE** on the SQL Warehouse that runs the load. If this was already granted during the ingestion setup, the same warehouse can be reused. Do not create the destination tables yourself. Unity Catalog gives a table's creator rights over it that these grants do not need to duplicate, and Birdie relies on that: every table this export manages has to be one Birdie's own identity created, or a later load can fail on ownership even with all of the grants above in place. ### Step 4 – Share the export configuration with Birdie Share the following details securely with Birdie: * Catalog name (e.g. `main`) * Export schema name (e.g. `birdie_export`) * **Staging volume name** (e.g. `birdie_stage`) — required; the export cannot stage files without it If not already shared during the data ingestion setup, also include: * Workspace host (e.g. `https://dbc-xxxx.cloud.databricks.com`) * SQL Warehouse ID * Service principal client ID and client secret, or the personal access token [Share credentials securely](/integrations-and-data-ingestion/securely-sharing-credentials-for-data-integration.md) with the Birdie team. Never send credentials via email or unencrypted channels. Birdie will configure the export connector and confirm once the integration is active. *** ### How the Export Works 1. **Staging upload:** each file is split into parts of at most 100,000 rows, converted to Parquet, and uploaded into a dedicated folder in the staging volume. Every run writes to its own folder, so a part left behind by an interrupted run is never loaded twice. 2. **Staging table:** the parts are loaded via `COPY INTO` into a temporary table, with every column as a string. 3. **Table creation:** on the first export of a file, the destination table is created from the columns the file carries. 4. **Load:** for incremental files, the rows whose primary keys the file carries are deleted and re-inserted, and every other row of the table stays untouched. Full files are replaced whole in a single statement. 5. **Cleanup:** the staged files are removed from the volume and the staging table is dropped. For incremental files, the delete and the insert run as two separate statements rather than one atomic merge, unlike Snowflake's export. A table read in between the two briefly shows that file's rows as missing. ### Exported Data Format Birdie creates one table per exported file. Table and column naming follows our standard CSV export conventions, so column names such as `Feedback ID` keep the spaces. The tables are created with [Delta column mapping](https://docs.databricks.com/en/delta/column-mapping) enabled, since Delta otherwise rejects column names containing spaces. If you read these tables from outside Databricks, your Delta reader needs to support column mapping. Every column is loaded as a string, except: * `Rating` and `Total Messages`, loaded as numeric * `Posted At`, `Ingested At` and `Updated At`, loaded as timestamps A value that cannot be read as that type is loaded as empty rather than failing the export. Each table also carries a `Birdie Exported At` column holding the time of the run that wrote the row. Data is de-duplicated on each load, so it will be ready to use. For details on joining tables, refer to our [data exports documentation](/admin-and-settings/export-import-csv.md#export). Some data (e.g., custom fields) may require additional handling due to their object-like nature. ### References * [Unity Catalog Privileges](https://docs.databricks.com/en/data-governance/unity-catalog/manage-privileges/privileges) * [Unity Catalog Volumes](https://docs.databricks.com/en/volumes/) * [COPY INTO](https://docs.databricks.com/en/sql/language-manual/delta-copy-into) * [Data Ingestion with Databricks](/integrations-and-data-ingestion/how-to-integrate-with/databricks.md) * [Data exports](/admin-and-settings/export-import-csv.md#export) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://ask.birdie.ai/integrations-and-data-ingestion/how-to-export-with/databricks.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.