Data Gateways
Overview
A Data Gateway is a lightweight application that is downloaded from within Data Hub to simplify cloud connectivity. It creates a persistent, outbound, two-way connection to Data Hub. The outbound connection eliminates the need to specify firewall rules in the environment hosting the Data Sources and expose the Data Sources publicly.
When a Data Model containing a Data Gateway connection is processed, the Gateway will:
Download the latest version of the Data Source
Open a connection to the Data Source
Compress and stream the requested data from the Data Source to Data Hub
Create a Data Gateway
Each Data Gateway is created with a unique key, allowing for the creation of multiple Data Gateways.
Host machine requirements:
Microsoft Windows Server edition 2016 or newer
Microsoft .Net Framework 4.8 (pre-installed on modern versions of Microsoft Windows)
Recommendations:
Two or more processor cores.
The Data Gateway service compresses data before upload. In some tests, it was found that installing the Data Gateway service on a 4 core server can result in a 30% performance gain.
Individual Data Sources can be added to a Data Gateway. If all of the Data Sources for a Data Model are accessible from a Data Gateway host machine, the Data Model can be added to the Data Gateway using this procedure.
Data gateway security
Connection security
The following connections are created by a Data Gateway:
Connection to Data Hub
Connections to Data Sources using the Data Gateway
MyGet for retrieving Data Hub Data Source packages
If Data Hub is running on an IIS server configured to use SSL or TLS, connections are secured by HTTPS using public certificates. This allows secure communication between Data Hub and the Data Gateway through an HTTPS tunnel, including the communication of connection credentials.
New and updated Data Sources are provided securely to the Data Gateway over HTTPS from a MyGet repository.
Credential storage security
Data Hub encrypts and stores model server resource credentials in the BI database.
Configure data gateway security
Firewall configuration
Data Hub and the Data Gateway require access to the following addresses through both the Microsoft Windows Firewall and public facing firewalls:
Incoming access for the External URL, as configured in the Data Hub portal settings (This connection should be mapped to Data Hub master node, allowing the Data Gateway to connect to the router service.)
Outgoing HTTPS access as outlined in the table
FQDN |
services.zapbi.com |
zapbi.myget.org |
mygetwwwzapbi.blob.core.windows.net |
s.sni.global.fastly.net |
Firewall considerations
When configuring the firewall to allow for communication between Data Hub and the Data Gateway, consider the following:
The required IP addresses presented above are subject to change. Allow the addresses to be resolved when creating firewall rules.
Data Hub and the Data Gateway require direct internet access through a firewall. They do not work through network proxies.
Only open the outbound ports that are required to connect to Data Sources.
General security
Credentials for the Data Gateway are stored in the Data Source resources of a Data Hub Model Server. This allows a single gateway to be used with multiple credentials.
Each set of credentials should only be allowed access to one Data Source (e.g. database and web service).
Access to the accounts used for connecting to a Data Source should be restricted.