S3 Imports

Import users, events, and auxiliary data from Amazon S3 buckets into MoEngage. Configure S3 integration for registered and anonymous user profiles.

Overview

MoEngage allows customers to import users and events through files stored inside S3 Buckets.

Types of Imports

MoEngage can import the following from your S3 buckets:

1. Registered Users: These are users who are already registered on MoEngage.
2. Anonymous Users: These are users who are not yet registered on MoEngage.
3. Events (Standard and User-Defined): MoEngage can import standard events like Campaign Interaction Events as well as your own user-defined events.
4. Auxiliary Data: These are users added temporarily.

Prepare the Files

Refer to the File Imports help doc to understand the supported file types, naming conventions, file structure, etc. Once you are ready with the files, you need to place them inside a folder on your S3 bucket.

Folder Structure

Place all files into an Amazon S3 Folder. You can configure the folder path while setting up the import.

Note that MoEngage will not look for files inside any subfolders.

Required Access Permissions

Before MoEngage can import files, grant it read access to your S3 bucket and set up a connection in the App Marketplace. For the IAM permissions and the connection setup, see Set Up Authentication and Grant Permissions in the Amazon S3 integration guide.

Set Up Imports from S3

Prerequisites

Make sure to complete all the steps mentioned in the File Based Imports help doc.

To set up the S3 imports, perform the following steps:

1. On the sidebar menu in MoEngage, hover over the Data menu item . The Data menu appears.
2. Click Data imports.
3. On the Data imports page, click the Data warehouses tab.
4. Click + Import in the upper-right corner and select Users or Events to create a new import.
5. Click the S3 tile.
6. Click Continue.

Step 1: Configure S3 Connection and File Format

User Imports

1. Select whether you want to import Registered users or Anonymous users.

2. Choose which user type you want to import. Both registered users import and anonymous users import expect different file names. Refer to the Naming Conventions section for more information.

Event Imports

1. For Event imports, select which event you want to import from the dropdown list.

2. Based on the event you pick, your file names must begin with the Event Name of your chosen event. The Select event dropdown displays the Display Names of all the available events in your MoEngage app.

In the above example, “App/Site Opened” is the Display Name of the chosen event and its Event Name is MOE_APP_OPENED, so the file names should begin with MOE_APP_OPENED.

3. You can also create a new event by clicking the + Create new event option at the end of the dropdown list.

4. Give your event a unique name. By default, your Display Name will be the same as the Event Name. You can go to the Data Management page to view or edit this event and your other MoEngage events. Note that any new events you create will only be available in your app after the first successful import.

Select Your S3 Bucket Connection

To select your S3 bucket connection, perform the following steps:

1. In the S3 connection list, click your preconfigured connection. If you have not created an S3 connection, click + Add connection to configure one in the App Marketplace. For more information, refer to Connect Amazon S3 in the App Marketplace.
2. In the File type list, click your file format (for example, click CSV). Only CSV and JSON files are supported.
3. In the Folder path box, type your directory path (for example, type bq_imports3). Do not include the bucket name or trailing slashes. MoEngage resolves your S3 bucket name automatically based on your marketplace connection.
4. Select the Is your file encrypted? check box if your source files are encrypted by using PGP. Paste your private PGP Decryption Key, attach your optional public Signing Key, and type your Key passphrase.

5. Enter a unique name for this import to easily identify it on the Imports Dashboard.

File Configurations

1. Select the DateTime format that will be in your file name. Your file name must end with _<date time format>.csv. Ensure you have at least one file available in the specified folder path; otherwise, you will not be able to continue. Click the “Fetch file(s)” button to fetch your latest file.

If your S3 credentials and chosen file format are correct, MoEngage displays a preview showing the top 5 rows of your file:

If MoEngage encounters an error while fetching the file from your S3 bucket, it displays one of the following alerts:

• Given S3 Access Key is not valid - Verify your S3 Marketplace connection details and try again.
• Please check the folder path and try again - Verify that your folder path is correct and ensure that compatible CSV files are present inside the folder.
• Found zero data rows in the file - The retrieved file does not contain any rows after the header row. MoEngage expects at least 2 rows (1 header row + 1 data row) in the file.

2. Once you are satisfied with the preview of the file, click Next to continue.

Step 2: Map Your Columns to MoEngage Attributes

Map the columns of your CSV file to the attributes present inside MoEngage:

1. Column name: The column name is mapped. Below the column name, MoEngage displays a sample value for your reference.
2. Map attribute: Select the target MoEngage attribute you want to map the column to. You can also click + Create attribute to define a new parameter.
3. More actions: Click Skip to exclude this column from being imported.

Ensure you configure the required mandatory mappings based on your import type:

1. User imports (Registered): You must map your identifier column to the MoEngage User ID attribute.
2. User imports (Anonymous): You must map your identifier column to the MoEngage Anonymous ID attribute.
3. Event imports: You must map your identifier column to the User ID attribute and map your timestamp column to the Event Time attribute.

To create a new attribute, click + Create attribute from the dropdown list:

Add Fields

You can also add fields directly from the dashboard by clicking on +Add Field to enter your attribute mappings manually.

Manifest Files

Optionally, you can choose to auto-map these columns by uploading a manifest file. To upload a manifest file, click the Upload mapping file option on the top-right of the mapping table:

Upload your manifest file and click Done. Your mappings will be auto-configured. For any additional attributes not currently present in MoEngage, a modal lists down these attributes for you to create:

Portfolio Support (Project-Level Routing)

Portfolio Routing

This feature allows you to route imported events from files stored inside your Amazon S3 bucket to specific projects within your MoEngage portfolio workspace using column mapping.

To route events to specific projects, map your S3 identifier column (such as brand_name or app_id) to the MoEngage attribute moe_project_name in Step 2 of the import configurations. Ensure that the source text values exactly match your MoEngage project names. Matching is case-sensitive.

• Successful routing: If the value matches your portfolio configuration, MoEngage ingests the user or event directly at that specific project level.
• Fallback behavior: If the mapping is missing, blank, or does not match a project name, MoEngage ingests the data at your global portfolio level.

Send Import Notifications

Select up to 10 emails to receive import status alerts when your import is created, succeeds, or fails.

When you are satisfied with all mappings, click Next.

Step 3: Select the Import Frequency

Define when MoEngage should retrieve your files:

1. One-Time Imports: Run the import immediately or schedule it for a future date and time.
2. Periodic Imports: Run your imports hourly, daily, weekly, or monthly.

Click Done when you are ready.

Duplicate Imports

An import is considered a duplicate when all of the following parameters are the same as an existing active import schedule:

1. Import Type: Users / Events
2. Import Sub-Type: Event Name / Registered / Anonymous
3. S3 Folder Path
4. Bucket Name

As long as any one parameter is different, the import is considered unique.

Frequently Asked Questions

My imports have failed. How do I check what went wrong?

Click the ellipsis icon on the right side of your import row and click View to review your details. Hover over the Failed status label to read the error details.

What happens when there is an error in fetching the files from the S3/SFTP folder?

MoEngage does not automatically retry imports when they fail. You can configure to receive an email alert upon failure by selecting the notification checkbox during setup.

A file failed with 'Applicable rate limit breached'. Will the periodic import reprocess it automatically?

No. When a file exhausts its processing retries and fails with the Applicable rate limit breached status, MoEngage cannot replay that file on its own, so the periodic schedule does not reprocess it automatically. This retry limit protects the ingestion pipeline from being overloaded. To reprocess the affected files, rename them and re-upload them to your S3 folder so that MoEngage treats them as new files. To reduce the chance of hitting rate limits, stagger large batches instead of scheduling many files at once (see Best Practices).

What if a scheduled import adds the data into a recently archived segment?

The new data is still added to the archived segment. You can unarchive the segment as required inside your Data Management dashboard.

How does MoEngage pick up new files in a periodic run?

During each scheduled run, MoEngage searches your designated folder path and selects new files based on your file prefix and datetime format configurations. MoEngage tracks file names and directory paths to ensure it only processes new files.

Will a periodic Data import pick up an earlier processed file?

No. MoEngage does not re-process files that have already been imported as long as the file names and directories remain identical. If you place a previously processed file inside a different folder, an import task targeting that new folder will process the file again.

Can a CSV file contain both User and Event data?

No. You must set up user imports and event imports as separate, distinct tasks inside MoEngage. Therefore, ensure that each source file contains only user attributes or event records.

Can an import be stopped while it's running?

Once an import process starts, it can’t be stopped midway. This is because the data goes through several steps, and interrupting it could lead to incomplete or inconsistent results. It’s best to let the current import finish processing.

How can I stop future runs of a scheduled import?

You can stop future scheduled imports from running automatically. To do this, find the import schedule on the Data Imports dashboard, click the ellipsis icon, and select the Archive option. This prevents the import from running on its next scheduled time.

What should I do if an import seems stuck or is taking a long time?

If an import appears to be stuck or is taking longer than usual, wait for the system to process the files. The system has checks in place to handle these situations automatically and will retry if necessary. Manually starting the same import again while it is still processing can cause conflicts and may prevent the original import from completing successfully.