Metadata-Version: 2.4
Name: webengage-migration
Version: 0.2.0
Summary: Webengage internal tool to migrate other party historical data into webengage ecosystem
Home-page: https://www.webengage.com/
Author: Nipun Patel
Author-email: nipunp27@gmail.com
License: MIT
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Requires-Dist: requests
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# WebEngage Migration Tool

A lightweight tool to migrate historical user or event data into WebEngage ecosystem.  Supports large datasets and can handle millions of rows, automatic batching, and adhere to WebEngage API rate limits.

---

## Features

- Handles large files efficiently (2M+ rows) - the file is streamed, never loaded fully into memory
- Uses the WebEngage **Bulk APIs** (`bulk-users`, `bulk-events`): 25 records per request, max 500 requests per minute
- Sends correct data types: string, number, boolean, array and map (JSON in a cell is sent as a real array / map)
- Retries rate limits (429), server errors (5xx) and network errors with backoff; isolates bad rows when a batch is rejected
- Validates every row before sending and prints a complete summary of what failed, on which row, and what was tried
- Writes failed rows to a ready-to-re-upload CSV; the input file is never modified
- Supports multiple datacenter routing (in, ksa, eug, us)

---

## Instructions

- Data should always be in **CSV** format (UTF-8) for both **events** and **users**. Row numbers in reports match the spreadsheet row (header = row 1).

- Columns are matched by **header name**, so column order does not matter. Empty cells are not sent.

    #### Sending Users

    - Standard fields (sent at the top level of the user): **userId**, **anonymousId**, **firstName**, **lastName**, **birthDate**, **gender**, **email**, **phone**, **company**, **city**, **region**, **country**, **emailOptIn**, **smsOptIn**, **whatsappOptIn**, **hashedEmail**, **hashedPhone**. Either **userId** or **anonymousId** is required.

    - **birthDate**: `yyyy-mm-dd`, `dd-mm-yyyy` or `yyyy-mm-ddTHH:MM:SS+hhmm`. **gender**: `male`, `female` or `other`. **phone**: E.164, e.g. `+917850009678`. Opt-in fields: `true` / `false`.

    - Every other column is sent as a **custom user attribute**.

    #### Sending Events

    - Standard fields: **userId** (or **anonymousId**), **eventName**, **eventTime**.

    - **eventTime** should follow this format: `yyyy-mm-ddTHH:MM:SS+hhmm` (eg: 1986-08-19T15:45:00-0800 or 2025-12-10T10:47:45+0530).

    - Every other column is sent inside **eventData**. WebEngage allows 25 event attributes of each data type; the tool warns if a file goes over.

    #### Data types

    Each custom column gets **one** data type for the whole file, so an attribute never arrives as two different types:

    | Cell values in the column | Sent as |
    |---|---|
    | all JSON starting with `[` | array |
    | all JSON starting with `{` | map (JSON object) |
    | all `true` / `false` | boolean |
    | all numbers | number |
    | anything else (including `+91...` phone numbers, `0123`, numbers longer than 15 digits, mixed values like `1` and `1,2`) | string |

    Use `--string-columns "col a,col b"` to force columns that look numeric (e.g. ids) to be sent as strings.

---

## Installation


To install the package run:

<pre style="font-size: 16px;">
pip install webengage-migration
</pre>

---

## Usage

To initiate the migration process run the below command:

<pre style="font-size: 16px;">
we --migrate {users/events} -f "{filename.csv}" -d {datacenter} -lc {license_code}
</pre>

<br>

| Argument           | Description                                           | Type                |
|--------------------|-------------------------------------------------------|------------------------------------|
| `{users/events}`   | Type of data you're migrating (`users` or `events`)  | ⚠️ Required                        |
| `-f "filename.csv"`| Path to the input CSV file                           | ⚠️ Required                        |
| `-d {datacenter}`  | WebEngage datacenter (`in`, `ksa`, `eug`, default is `us`) | ⚙️ Optional  |
| `-lc {license_code}`| Your WebEngage license code                         | ⚠️ Required                        |
| `--dry-run`        | Validate the file and print the first payload; nothing is sent | ⚙️ Optional |
| `--string-columns` | Comma separated columns to always send as strings     | ⚙️ Optional |
| `--rpm`            | Requests per minute, max (and default) 500            | ⚙️ Optional |

<br>

**Example:**


**Send users**: `we --migrate users -f "datafile.csv" -d in -lc in~~1234c456`

**Send events**: `we --migrate events -f "datafile.csv" -d in -lc in~~1234c456`

**Check a file first**: `we --migrate events -f "datafile.csv" -d in -lc in~~1234c456 --dry-run`

<br>

Once migration is initiated, provide the **API key for authorization** (input is hidden; you can also set the `WE_API_KEY` environment variable), **API key** will be found in **webengage dashboard for provided license code** under:

***Data Platforms → Integrations → Rest API → `API KEY`*** 

![Usage Example](https://logextractor.s3.ap-south-1.amazonaws.com/apikey.png)

---


## WebEngage API Rate Limits

WebEngage limits bulk event / users API usage to:

- **25 events / users** per API request
- **500 requests** per minute (Total **12,500 events** per minute)

The tool enforces this with a sliding 60-second window that also counts retries, and pauses when WebEngage answers `429` (honouring `Retry-After`).

---

## Failures, retries and re-upload

| What happened | What the tool does | Reported as |
|---|---|---|
| Row fails validation (empty userId / eventName, bad eventTime / birthDate / gender, invalid JSON) | Not sent | `invalid_payload` |
| `429` rate limit, `5xx`, `408`, timeout or network error | Retried up to 5 times with backoff, using the same `x-request-id` so WebEngage drops duplicates | `queued_after_retry`, or `retries_exhausted` |
| Batch refused with another `4xx` (e.g. `400`) | Every row of the batch is re-sent on its own to find the bad rows | `rejected_by_api` (with the API message) |
| `401` / `403` / `404` (wrong API key, license code or datacenter), or Ctrl+C | Run stops; unsent rows are kept | `not_sent` |
| String value of 1000+ characters | Sent, flagged | `warning_sent` |

After every run the console shows a summary (counts, failed rows with reason, HTTP status and attempts). When anything failed, two files are written next to the input:

- `<file>.failed.csv` - the failed rows with the original header. Fix them and run the same command on this file to re-upload only those rows.
- `<file>.upload_report.csv` - one line per row with `row, result, http_status, attempts, reason, what_was_tried`.

---

## Legal Notice

This tool is an **internal property** of **WebEngage** and is strictly for **migration purposes**. It is owned by **Nipun Patel (Copyright)** and any misuse, unauthorized distribution, or external sharing will lead to **legal consequences**.

---

© WebEngage. All rights reserved.
