Incremental updates

Overview

An incremental update is the process of discovering recent changes in your ticketing system and then applying those same changes to KORE’s staging database. This allows us to keep our copy of your database up to date without needing to redownload the entire set of data. We then process the copied data (including deduplication) and push the updates to your CRM.

None of the ticketing systems we support allow for real-time monitoring. Instead, we check for updates hourly. We copy the raw data into our staging area, then apply the updates to our staging database. The updated data is then processed and deduplicated so that it’s available through KORE Ticketing or KORE Data Warehouse & Analytics.

This process applies only to syncing data from a ticketing system, not into it. KORE Ticketing only writes data to a ticketing system when you make changes using the Ticketing Data Manager (TDM).

Three-phase update

Incremental updates occur in three phases: Download, Apply, and Push.

In the Download phase, we send a request to the ticketing system for all records which have changed since the last time we got an update. During the Apply phase, we apply those downloaded records to our staging database. In the Push phase, we process the data (including deduplication) and push the updates to the KORE Ticketing database that your CRM uses.

Incremental updates typically begin hourly. When one phase completes successfully, it kicks off the next phase. A normal incremental update looks like this:

Delays

We start an incremental update hourly unless a Download from the previous update is still in progress. This could happen if there’s an unusually large amount of data or if the systems involved are running slowly. If that happens, we skip the hourly update so the systems can catch up.

Upon successful completion of the Download phase, we kick off the Apply phase. However, only one Apply phase can run at a time. (This helps ensure that data is processed in chronological order. If the data were processed out of order, it could cause newer data to be overwritten with older data.) If an Apply phase is still in progress from the previous cycle, the data we just finished downloading can’t be applied to the staging database until the next time a Download phase kicks off an Apply phase.

When the Apply phase finishes, we then kick off the Push phase. The Push phase deduplicates contacts, assigns ticketing groups, and performs other processing. It then copies the processed data into the KORE Ticketing database that your CRM uses. Like the other phases, only one Push can be in progress at a time. If a previous Push is still in progress, then the data applied to the staging database won’t be pushed into the KORE Ticketing database until another Apply phase completes and kicks off a new Push phase.

Error recovery

The three phases are designed to ensure that no data is lost if there’s a problem anywhere in the update chain.

1. Download: We keep a timestamp of when the last successful download began. When an hourly Download phase begins, we request all the updates made since that time. If a Download phase fails, the next hourly Download phase simply uses that same timestamp for its request.
2. Apply: A download contains many independent updates, each of which has its own timestamp representing when the ticketing system processed it. During the Apply phase, we apply each of these updates to our staging database in chronological order. When an update is successfully applied, we copy that update’s timestamp so we can track the most recent successful update to the staging database. If an Apply phase fails, the next Apply phase will begin applying data from that timestamp.
3. Push: Each individual update applied to the staging database now goes through deduplication and other processing. Each of those updates has its own timestamp, so we track the timestamp of the last successfully pushed update. If a Push phase fails, the next Push phase begins pushing data from that timestamp.

Most errors are temporary and resolve themselves during the next attempt. But if a phase fails repeatedly, our monitoring system alerts KORE engineers to the problem.

The following diagram illustrates how the three-phase update system recovers from various problems:

In this example, Download Batch 2 can’t kick off an Apply phase since one is already in progress. When Download Batch 3 fails, no Apply phase is kicked off, so Download Batch 2 remains unapplied and there’s no retry of Push Batch 1 yet. In the following hourly cycle, the new Download Batch 3 contains everything from the original Download Batch 3 plus the additional updates made in the ticketing system after the failed download began. When that Download phase succeeds, the previously queued batches are included in the Apply and Push phases.