Skip to main content

Download service

KORE Team
  • Updated

Overview

KORE has designed and built a data delivery service which can be installed to on-premise servers and machines which will enable extraction, packaging and delivery of data to KORE's servers for ingestion to the data warehouse. The intent of this service is to fulfill the need for clients to have data extracted from systems that teams have access to but that have no automated data delivery method, or when the delivery method provided does not meet the requirements of the project the team wishes to accomplish. 

Solution architecture

An overview of the download service’s architecture is described here. It requires limited access to 3 AWS services in the cloud in order to transfer data from the source environment to KORE’s cloud network.

 

Requirements

Here’s what’s required to install and operate the download service

Server requirements

These are the minimum server requirements for the service. Actual requirements may exceed the minimum particularly in data volume exceeds server capacity

  • Operating System

    • Windows 8 or above. Window Server 2008 or above.

  • Processor

    • 2.0 GHz processor speed for data extraction process

  • Random access memory (RAM)

    • Minimum 4 GB memory to run the service

    • 8-16 GB recommended for larger data extraction processes

  • Hard disc drive (HDD)

    • 500 MB permanent storage for the service installation

    • 10 - 20 GB available storage for the data extraction process. This will be freed each time the process has completed.

Environment requirements

Network

  • Minimum 10 MBPS network upload speed to handle the data transfer to KORE's cloud data storage services

  • Outbound network traffic on port 443 (HTTPS) for connectivity to 3 AWS services (S3, System Manager, DynamoDB)
    Note: Outbound traffic can be restricted to an allow listed IP address if additional network security is required, please reach out to KORE for this information.

  • Inbound is not required

Schedule

  • Ability to schedule the execution of the application
    Note: KORE recommends task scheduler or CRON jobs

.NET CORE Runtime

  • Version 3.1 or great is required

Maintenance

  • Server will be configured, operated and maintained by the team.
    Note: Initial installation and version upgrades will be handled with conjunction with a KORE technical resource

Database connectivity

  • Read only connectivity between the service installation server and the source database.
    Note: KORE recommends confirming this through an ODBC connection on the installation server

Database requirements

The following requirements are necessary for the source database.

User

  • a read only database user is required to access the source database when the service runs

Permissions

  • it requires access to any database object (table, view, stored procedure, function etc) required to extract the data

  • it requires access to the system information for tables for extracting metadata (this varies from vendor to vendor but normally this is found in INFORMATION_SCHEMA.tables)

Extract time period

  • a dedicated time period is required where the service is able to run to extract the data.
    Note: the length of the time period varies depending on the volume of data, we recommend scheduling this time period outside of normal business hours

Staging area (optional)

  • A staging area where the service can temporarily stage or prepare the data in the database may be required for complex data extraction processes
    Note: this can be achieved by a few different options, KORE will provide the various options so we can pick the best solution for your implementation

Database replica (optional)

  • KORE recommends providing connectivity to a replicated or copy of the database rather than the primary or source database to prevent any potential impact to production database.

Business requirements

The customer is responsible for the on going maintenance of the service, including

  • Monitoring and maintaining the service scheduler

  • Server maintenance (upgrades, updates, patches etc)

  • Managing system requirements on the server (minimum requirements of CPU, RAM, HDD etc)

  • Database credentials (user management, password rotation)

  • Updating KORE provided credentials (when requested to be updated by KORE)

  • At REST encryption for stored credentials

  • Monitoring the extract time period

  • Troubleshooting and diagnosing issues when they arise (a KORE technical resource can assist with this)

Pre-installation checklist

These steps must be completed by the vendor or customer prior to installation of the download service.

Steps

  1. Provision server (new or existing) for the service installation

  2. Create a read-only user

  3. Confirm connectivity to the source database

  4. Whitelist Proxy IP (optional)

  5. Validate connectivity to AWS services

Provision server

Please confirm you have a server available which meets the minimum requirements defined above.

Create read-only user

Please create a new database user which will be used by the download service to access the source database. You will need to refer to the documentation specific to your database vendor on how to create this user.

The user should meeting the following requirements

  • have read only permissions to the database objects required to send to KORE

  • have read only access to the information_systems.tables system view

  • have the ability to export data to the local disc

Confirm connectivity to the source database

Confirm connectivity to the source database with the user which was created in the step above. To do this KORE recommends creating an ODBC connection to confirm connectivity

Before you progress with this step, please confirm you have installed the latest ODBC driver for your source database.

Note: these steps will different slightly depending on the vendor of the source database you are trying to connect to, this example uses Amazon Redshift

  1. Open ODBC → System DSN → Add

     

  2. Enter the connection details for the source database. Here is an example of

     

  3. Test connectivity

Allowlist Proxy IP

If you require to restrict inbound and/or outbound traffic to a specific IP address, please ensure you have added the following IP and port to your Allowlist in your network

  • IP address: 54.212.177.129

  • Port: 443

Validate connectivity to AWS services

The last step is to confirm you have connectivity to the necessary AWS services used by the download service. Please reach to KORE prior to attempting this so we can validate your required network configuration and specific requirements.