Data Gateways

Overview

A Data Gateway is a lightweight application that is downloaded from within Data Hub to simplify cloud connectivity. It creates a persistent, outbound, two-way connection to Data Hub. The outbound connection eliminates the need to specify firewall rules in the environment hosting the Data Sources and expose the Data Sources publicly.

When a Data Model containing a Data Gateway connection is processed, the Gateway will:

  • Download the latest version of the Data Source

  • Open a connection to the Data Source

  • Compress and stream the requested data from the Data Source to  Data Hub

Create a Data Gateway

Each Data Gateway is created with a unique key, allowing for the creation of multiple Data Gateways.

Host machine requirements:

  • Microsoft .Net Framework 4.8 (pre-installed on modern versions of Microsoft Windows)

and either of the following:

  • Microsoft Windows Server edition 2012 R2 or newer

  • Microsoft Windows 7 or newer

Recommendations:

  • Two or more processor cores.

  • The Data Gateway service compresses data before upload. In some tests, it was found that installing the Data Gateway service on a 4 core server can result in a 30% performance gain.

  • Individual Data Sources can be added to a Data Gateway. If all of the Data Sources for a Data Model are accessible from a Data Gateway host machine, the Data Model can be added to the Data Gateway using this procedure.

Data gateway security

Connection security

The following connections are created by a Data Gateway:

  • Connection to  Data Hub

  • Connections to Data Sources using the Data Gateway

  • MyGet for retrieving  Data Hub Data Source packages

If  Data Hub is running on an IIS server configured to use SSL or TLS, connections are secured by HTTPS using public certificates. This allows secure communication between  Data Hub and the Data Gateway through an HTTPS tunnel, including the communication of connection credentials.

New and updated Data Sources are provided securely to the Data Gateway over HTTPS from a MyGet repository.

Credential storage security

Data Hub encrypts and stores model server resource credentials in the BI database.

Configure data gateway security

Firewall configuration

Data Hub and the Data Gateway require access to the following addresses through both the Microsoft Windows Firewall and public facing firewalls:

  • Incoming access for the External URL, as configured in the  Data Hub portal settings (This connection should be mapped to  Data Hub master node, allowing the Data Gateway to connect to the router service.)

  • Outgoing HTTPS access as outlined in the table

IP

FQDN

168.61.152.29

services.zapbi.com

40.76.66.97

zapbi.myget.org

52.239.152.234

mygetwwwzapbi.blob.core.windows.net

Firewall considerations

When configuring the firewall to allow for communication between  Data Hub and the Data Gateway, consider the following:

  • The required IP addresses presented above are subject to change. Allow the addresses to be resolved when creating firewall rules.

  • Data Hub and the Data Gateway require direct internet access through a firewall. They do not work through network proxies.

  • Only open the outbound ports that are required to connect to Data Sources.

General security

  • Credentials for the Data Gateway are stored in the Data Source resources of a  Data Hub Model Server. This allows a single gateway to be used with multiple credentials.

  • Each set of credentials should only be allowed access to one Data Source (e.g. database and web service).

  • Access to the accounts used for connecting to a Data Source should be restricted.