Databricks Exports

Export campaign interaction events from MoEngage to Databricks using a direct export or OpenSharing (zero-copy sharing). Configure either method for scalable event analytics.

This feature is part of the Streams paid add-on. To check what’s available for your plan, refer to Plans and Pricing. To enable it for your account, contact your MoEngage Customer Success Manager (CSM) or the Support team.

Overview

Databricks is a platform that allows organizations to store, analyze, and process large volumes of structured and semi-structured data in a highly scalable and efficient manner. With its unique architecture, organizations can consolidate their data, perform quick analytics, and gain valuable data-driven insights accessible to all users.

How Does MoEngage Work with Databricks?

With this integration, you can export data from campaign interaction events to your Databricks tables.

Delivery Methods

MoEngage delivers event data to Databricks in two ways:

• OpenSharing (Zero-Copy Sharing): Your workspace queries MoEngage-hosted data in place through the Delta Sharing protocol. This is the recommended method.
• Direct export: MoEngage writes events into a table in your own schema. Use this method when your workspace does not have Unity Catalog enabled.

Both methods produce the same table schema and refresh hourly. The following table compares them:

	OpenSharing (Zero-Copy Sharing)	Direct Export
How data arrives	Your workspace queries MoEngage-hosted data live through the Delta Sharing protocol.	MoEngage writes events into a table in your schema.
Credentials MoEngage needs	None. You provide a sharing identifier, which is not a secret.	WRITE access to your Databricks catalog and schema.
IP allowlisting	Not required.	Required if your system enforces it.
Where data is stored	In a dedicated MoEngage-hosted catalog that you mount.	In your own catalog and schema.
Requires Unity Catalog	Yes.	No.
Setup time	Varies. Setup can take weeks or longer, depending on your workspace setup and enablement.	Self-serve on the App Marketplace.

The two methods use separate storage layers. Records already delivered through a direct export do not appear in an OpenSharing catalog, and vice versa. To move between methods, contact your MoEngage Customer Success Manager (CSM). OpenSharing setup may take a few weeks or more, depending on your workspace setup, so plan accordingly.

Integration

Prerequisites

The prerequisites depend on the delivery method you choose. Click the tab for your method:

OpenSharing (Zero-Copy Sharing)

• Ensure your Databricks workspace has Unity Catalog enabled.
• Retrieve your sharing identifier, formatted as <cloud>:<region>:<metastore-id>. For more information, refer to Step 1: Request Enablement.

Direct Export

• Ensure you have a Databricks account with relevant permissions to create/edit Databricks instances, roles, and policies. It is optional to create a new user; you must have access to create a new user in your Databricks database.
• If your system requires IP whitelisting, you must whitelist MoEngage IPs. For more information, refer here.
• Ensure you have a Databricks Connection on the App marketplace.

Set Up Your Export

• For the full list of events and attributes, refer to the Data Export Glossary.
• The event_time of the exported events is always in UTC.
• MoEngage does not support exporting historical data. Data flows from the point you enable the export.

The setup steps differ by delivery method. Click the tab for the method you are using:

OpenSharing (Zero-Copy Sharing)

With OpenSharing, your workspace queries MoEngage-hosted data directly through the Delta Sharing protocol instead of receiving file-based exports. You do not exchange credentials or write tokens, and your queries always read the current data. You query the shared catalog with your existing SQL editors, notebooks, and BI dashboards through standard Unity Catalog workflows.

The following diagram shows how data moves between MoEngage and your workspace:

Step 1: Request Enablement

To request OpenSharing, perform the following steps:

1. In your Databricks workspace sidebar, click Catalog.

2. Click Share, and then click OpenSharing.

   

3. On the OpenSharing page, click Sharing ID to copy your sharing identifier.

   

Send the sharing identifier to your MoEngage CSM.

Step 2: Accept the Share

MoEngage usually provisions the share within seven business days. Your CSM then sends you the name of your data share.

To accept the share, perform the following steps:

1. In your Databricks workspace, click Catalog > OpenSharing, and then click the Shared with me tab.

   

2. Click the MoEngage provider that matches the name your CSM sent you.

3. Next to the share, click Mount to catalog.

   

4. Enter a catalog name, for example, moengage_shared_data, and then confirm it.

Step 3: Configure Your Export

To configure the export in MoEngage, perform the following steps:

1. On the left navigation menu in the MoEngage dashboard, click App marketplace.
2. On the App marketplace page, search for Databricks, and then click the Databricks tile.
3. Click the Integrate tab.

Create the export as described in Add a New Databricks Export on the MoEngage App Marketplace. In the Export into schema list, select the shared catalog’s schema. The list shows the same value your CSM shared with you.

OpenSharing provisions the connection for you, so MoEngage does not prompt you for a connection, service principal credentials, or write access.

Step 4: Query Your Data

The first rows can take up to 30 minutes to appear in your shared catalog. To verify the data, perform the following steps:

1. Run the following query:

   SELECT * FROM <your_catalog_name>.<schema>.<table_name>;
   

   Replace the placeholders with your catalog, schema, and table names.

2. Click Catalog Explorer.

   

3. Click a table, and then click Sample Data.

Catalog Explorer lists the shared catalog under Shares received, along with its schemas and tables. The Sample Data view previews records and attribute definitions.

If you want MoEngage to walk you through the schema, request a technical schema review from your CSM.

Step 5: Manage Access Within Your Organization

Because the shared catalog sits in your own infrastructure, you control access through Unity Catalog. Assign the USE CATALOG and SELECT privileges to the users or groups that need the data.

MoEngage has no write access to the catalog, and either party can end the connection at any time. Tell your CSM about any changes to your workspace setup or sharing identifier, because these changes can interrupt delivery.

Direct Export

Step 1: Grant Your User’s Role the Required Permissions

MoEngage needs WRITE access to your Databricks database to store data. You can grant these permissions to an existing database user or create a new database user for MoEngage.

Use the following code to configure write access for data storage in your Databricks database:

GRANT USE CATALOG ON CATALOG `<CATALOG_NAME>` TO `<EMAIL_ID_OF_THE_USER>`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA `<CATALOG_NAME>`.`<SCHEMA_NAME>` TO `<EMAIL_ID_OF_THE_USER>`;

Change the following values:

• <CATALOG_NAME> - The name of your catalog.
• <SCHEMA_NAME> - The name of your database or schema.
• <EMAIL_ID_OF_THE_USER> - The email ID of the user who created the token.

Step 2: Use a Unity Catalog Volume for Faster Exports (Recommended)

This step is optional. MoEngage can use a Unity Catalog volume to export data more efficiently. To use this method, create a dedicated volume and grant MoEngage access to it.

Use the following code to create the volume and grant access:

-- Create the volume. This is a one-time activity, and Databricks provisions the storage.
CREATE VOLUME IF NOT EXISTS <CATALOG_NAME>.<SCHEMA_NAME>.<VOLUME_NAME>;

-- Grant MoEngage read and write access on the volume.
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <CATALOG_NAME>.<SCHEMA_NAME>.<VOLUME_NAME>
  TO `<GRANTEE>`;

Change the following values:

• <CATALOG_NAME> - The name of your catalog.
• <SCHEMA_NAME> - The name of your database or schema.
• <VOLUME_NAME> - The name of the volume you are creating.
• <GRANTEE> - The service principal’s application ID (if using OAuth or a Service Principal PAT) or the user’s email address (if using a User PAT).

Enter the volume name in the Unity Catalog Volume field when you add the export.

Step 3: Add the Export on the App Marketplace

Continue with Add a New Databricks Export on the MoEngage App Marketplace, below.

Add a New Databricks Export on the MoEngage App Marketplace

Both delivery methods use this step. With OpenSharing, the connection already exists, so you only choose the schema, events, and attributes.

To add a new Databricks export, perform the following steps:

1. On the left navigation menu in the MoEngage dashboard, click App marketplace.

2. On the App marketplace page, search for Databricks.

   

3. Click the Databricks tile.

4. On the Databricks page, click the Integrate tab, and then click + Add Export.

5. Enter the following details:

   Field	Required	Description
   Export name	Yes	Type a name for the Databricks export.
   Databricks connection	Yes	Select your Databricks connection.
   Export into schema	Yes	Select the schema that MoEngage uses to create new tables and export your events into.
   Unity Catalog Volume	Optional	Enter the name of the volume you created for MoEngage. Leave it empty if you are not using a volume. To create one and grant MoEngage access, refer to the Direct Export tab under Set Up Your Export.
   User properties	Optional	Select the user properties (user properties and device properties) you want to export with your events.
   Select events	Yes	Select the events you want to export to Databricks. You can choose to export all events, including existing and new ones coming into MoEngage, or manually select specific events to export.

   

6. Click Save.

Export Process and Frequency

Both delivery methods refresh hourly, and neither requires you to run manual sync jobs.

For a direct export, MoEngage creates a destination table in your schema after you save the export, and then appends new event data at hourly intervals:

• Initial action: MoEngage creates the table when the first export is triggered.
• Recurring action: MoEngage appends data to the same table every hour.

For OpenSharing, MoEngage refreshes the shared datasets every hour. Your queries read the current state of the data, so there is no table for MoEngage to write into on your side.

Sample Table Structure

MoEngage creates a single table using the following format:
moe_events_<table_creation_timestamp>.

For a direct export, this table is created in your schema. For OpenSharing, it appears in the shared catalog you mounted.

This table is created per connection. All hourly exports and different event types configured under this connection are consolidated and appended to this specific table.

The overall structure of the table remains fixed for all event types. Any new user property or event property that is exported is included in the user_attributes or event_attributes column, respectively. All events are consolidated in a single table containing necessary information, such as event names and event times, for easy analysis:

Field	Type	NULL
app_name	VARCHAR	NOT NULL
event_name	VARCHAR	NOT NULL
event_code	VARCHAR	NOT NULL
event_uuid	VARCHAR	NOT NULL
event_time	TIMESTAMP	NOT NULL
event_type	VARCHAR	NOT NULL
event_source	VARCHAR	NOT NULL
uid	VARCHAR	NULL
user_attributes_moengage_user_id	VARCHAR	NULL
device_attributes_moengage_device_id	VARCHAR	NULL
push_id	VARCHAR	NULL
email_address	VARCHAR	NULL
mobile_number	VARCHAR	NULL
event_attributes_app_version	VARCHAR	NULL
event_attributes_sdk_version	VARCHAR	NULL
event_attributes_platform	VARCHAR	NULL
event_attributes_campaign_id	VARCHAR	NULL
event_attributes_campaign_name	VARCHAR	NULL
event_attributes_campaign_type	VARCHAR	NULL
event_attributes_campaign_channel	VARCHAR	NULL
event_attributes_readable_campaign_id	VARCHAR	NULL
event_attributes_parent_campaign_id	VARCHAR	NULL
event_attributes_parent_flow_id	VARCHAR	NULL
event_attributes_parent_flow_name	VARCHAR	NULL
event_attributes_variation_id	VARCHAR	NULL
event_attributes_locale_id	VARCHAR	NULL
event_attributes_locale_name	VARCHAR	NULL
event_attributes_url	VARCHAR	NULL
event_attributes_timestamp	TIMESTAMP	NULL
event_attributes_first_session	BOOLEAN	NULL
event_attributes_logged_in_status	VARCHAR	NULL
event_attributes	VARIANT	NULL
user_attributes	VARIANT	NULL
device_attributes	VARIANT	NULL

The schema is the same for both delivery methods. Custom column naming is not supported.

Export Failure Policy

MoEngage pauses an export that keeps failing. While it is paused, MoEngage collects the events the export would have delivered, and you replay them after you fix the problem.

When MoEngage Pauses an Export

Exports to Databricks run hourly. MoEngage will retry an export multiple times before marking it as failed. The export will be paused after 3 consecutive scheduled export failures.

When MoEngage pauses an export, it alerts you and shows the export as Paused on the App Marketplace.

What Happens While an Export Is Paused

MoEngage collects the events the export would have delivered and keeps only the most recent 7 days. If the export stays paused for longer than a week, MoEngage drops the oldest days as newer ones come in.

If an export pauses more than once before you replay, MoEngage combines the data from all those pauses, provided it falls within the 7-day window.

Resume an Export

Find the Failure Reason

On the App Marketplace, click the Integrate tab, and then hover over the Paused label on the export. MoEngage names the reason for the failure, such as a missing write permission on the destination table, and tells you how to fix it.

Enable the Export

Fix the cause at your end, and then turn on Export status for the export. Do this as soon as you can, because the 7-day window keeps moving while the export stays paused.

When you enable the export, MoEngage stops collecting and sets aside what it has: the last 7 days of paused data, or all of it if the pause was shorter. This is the data you can replay. It stays fixed from then on, and does not reduce further even if you wait to replay it.

Replay Paused Data

Request a Replay

MoEngage saves the data it collected while the export was paused. An export with data waiting shows a Replay available label on the Integrate tab.

To replay, perform the following steps:

1. Hover over the Replay available label, or open the kebab menu on the export.

2. Click Replay paused data.

   

3. In the Replay paused data dialog box, click Confirm.

   

A replay sends everything MoEngage collected while the export was paused. MoEngage appends the data to the destination table.

You can replay at any time until 30 days from the date the oldest paused data was collected. After that, the data is no longer available.

Replay before the export pauses again. If it pauses a second time before you replay, MoEngage adds the new paused data to the data still waiting from the first pause. The combined data is then trimmed back to the most recent 7 days, so if the two pauses are more than 7 days apart, the older data is lost.

For example, your export pauses on day 1. You enable it but do not replay. It pauses again on day 20. Since that is 19 days later, the day 1 data falls outside the 7 day window and is gone, even though its 30 day replay window had not expired. If the second pause had happened within 7 days of the first, the day 1 data would still be included.

Replay Results

When the replay finishes, MoEngage emails the user who requested it with one of the following results:

Result	What it means
Successful	MoEngage delivered all the collected data. The replay option is no longer available.
Partially Successful	MoEngage delivered some of the collected data, and then hit an error. The replay stops and MoEngage pauses the export again. The email states why the replay did not finish. Fix the cause, enable the export, and then replay again. For the data you can replay, refer to What Happens While an Export Is Paused.
Failed	MoEngage delivered none of the collected data, because of an error. The replay stops and MoEngage pauses the export again. The email states why the replay did not finish. Fix the cause, enable the export, and then replay again. For the data you can replay, refer to What Happens While an Export Is Paused.

You can run only one replay per export at a time.

Avoid Duplicate Data

Every replay sends the whole collection again, because MoEngage does not deduplicate. If you replay more than once, deduplicate at your end on the event_uuid field, which is unique for each event. For the fields in the export, refer to the Data Export Glossary.

Export Paused Alerts

MoEngage alerts you when it pauses an export, so you can fix the issue and replay the data collected while the export was paused.

MoEngage subscribes workspace admins to the Data Exports system alert by default, and adds the user who creates or updates an export. To subscribe or unsubscribe anyone else, go to the Alert Management page. Subscribing or unsubscribing changes notifications only for that person, and does not affect anyone else’s subscription.

Frequently Asked Questions

What does the Replay available label mean?

MoEngage is holding data it collected while the export was paused. Hover over the label or open the kebab menu, and then click Replay paused data.

My export was paused for three weeks. Why can I replay only part of it?

MoEngage keeps only the most recent 7 days of paused data, so a three-week pause leaves only the last 7 days to replay.

Why did my replay come back as Partially Successful?

MoEngage delivered some of the data and not the rest, usually because the destination rejected part of it. Fix the cause and replay again. The next replay sends the whole collection, not only the part that failed.

Can I run two replays at the same time?

No. You can run only one replay per export at a time. Wait for the result email before you request another.

I replayed twice and I see duplicate rows. Why?

Every replay sends the whole collection again, because MoEngage does not deduplicate. Deduplicate at your end on the event_uuid field, which is unique for each event.

How soon will I start getting the data?

It typically takes up to 30 minutes for the automated data exports to be set up for your account and for you to start seeing data there.

Can I export additional events at a later time?

Yes, if you need to export new events at a later time, you can always edit your configuration on the App Marketplace and get the new events added to your exports.

Can I export historical data using Automated Data Exports?

As of now, you cannot export historical data using Automated Data Exports. Once configured, you will start seeing data for each event from the time you enabled your exports.

Is there any pricing involved for Automated Data Exports?

If your billing plan does not cover this already and you need to get this enabled at a later time, you would need to work with your Customer Success Manager to get this included in your billing plan. Based on the amount of data that MoEngage will place in your Data warehouse/Partner/API on a monthly basis, this could lead to additional charges.

Does Data Exports guarantee the latest user properties?

No, Data Exports is primarily built to export your Events in near real-time and user properties in MoEngage are updated asynchronously so it is currently not possible to guarantee the latest values of user properties in the exports.

Can I move from a direct export to OpenSharing?

Yes. Contact your MoEngage CSM to change delivery methods. The two methods use the same schema but separate storage layers, so records already delivered through your direct export do not appear in the OpenSharing catalog.

Can I rename the columns or tables in the OpenSharing catalog?

No. The schema is standardized and custom naming is not supported. You choose which events and attributes to export during configuration, as described in Step 3: Configure Your Export.

Who do I contact after the OpenSharing connection is live?

Your MoEngage CSM is the primary contact for maintaining the connection, resolving technical questions, and adjusting your data requirements.