Data Hub 10.2 on-premises release notes

While processing a model, the progress has always been visible on the model properties panel while processing. Once the processing has completed the progress tree isn't available anymore.

A new addition has been made to the background task screen. A Model task log is now available on each model process task, while running but also as history, so it can be revisited at any time.

Configurable data source migration retries during model processing were further enhanced to also include retrying when duplicate key exceptions are experienced.

Processing selected amounts of source data has become increasingly important for customers with large data sets. An exciting enhancement has been added to assist in configuring subsets of source data to migrate.

Partitioned pipeline processing provides the ability to migrate only periods of active source information based on date partitions, using a suitable date column, instead or reloading a full data set.

For example a large transaction table can be set to a Partition period of Month, and Active partitions of 4 which will process all data for the four months leading up to the current date, including the month of the current date. If current date is 15 June, all data from 1 March to 15 June will be migrated. Detailed information here

Configuration for this feature can be set in the Incremental Process (Source) section of individual pipelines.

Configurable historical partition threshold

Configure Empty Bulk Historical threshold: Sparsely distributed data may pre-emptively trigger the bulk historical partition, particularly when partitions are configured to be a day in size. This setting raises the threshold for a bulk load from the default single empty partition.

Per company and per tagged pipelines processing are new additions to model processing. It gives users the ability to process their models only for a specified company or group of tagged pipelines within the Data model, and not necessarily processing the entire model.

Pipeline tagging is achieved by adding labels in the tags column on the model screen pipeline grid, to logically group pipelines together. More information about Modules and tags

This feature becomes very useful to organizations that have several companies and/or large data sets. The enhancement can impact performance positively by reducing the amount of data to refresh, and therefore reduce processing time and removing data that is not required at any given point in time. Detailed information can be found in Processing parts of a model.

Process per company and process per tag have also been incorporated in a new version of the Model Process API

Both these new settings are available in two areas:

During processing, relying on dependencies sometimes inflated processing times and it wasn't clear how efficient the processing of individual pipelines was.

Wait times for dependencies are now excluded from pipelines processing time calculations, so users can gain a more accurate understanding of the actual processing times and a clearer picture of the efficiency or potential bottlenecks.

The ability to freeze the column headers on the preview pane has been added.

Notifications for model processing warnings have been improved. Process warnings and errors can be seen in the background tasks and when analytics are opened that connect to a cube that has processing warnings, a relevant notification will be shown on the alerts icon.

Inside the Column and Lookup dropdowns on pipelines, the respective Apply and Add buttons have been moved outside the scrollable area, so that it is always visible.

A new sql expression category Utility Functions was added for improved grouping and organization of functions.

Previously the only way to refresh the table list on a union step, was to use the refresh schema button on the Initial step.

A new Refresh Union button has been added to the union step to refresh the schema of only the unioned tables.

When previously selected companies in a data source becomes unavailable or disconnected, it will now show a validation warning for improved awareness.

Marking a company as a primary company is being removed from the following data sources:

The date the datasource was last updated is now visible in the version section of the datasource properties panel.

All fields on the resource properties screen are selectable as appropriate.

Background tasks now shows a more helpful date and time of actual execution.

Pipeline and column physical name behavior has been improved. Datasources like Dynamics, SAP, SYSPRO and Sage that provide captions for table and column names in the source.

In the past, pipeline and column physical names were derived from the table name.

The changed behavior will now default to using the caption for the physical name if it is available.

A new Excel Online connector has been introduced. The connector uses Azure AD authentication to connect to a SharePoint site. It reads and can digest all excel workbooks on a given SharePoint site or document directory.

Performance: When using the 64bit ODBC driver, slow performance and crashing of the driver can occur when a result set returned is too large. A pagination strategy has been implemented to improve this by allowing configuration of the following settings. Read more about the strategy here

Performance fallback: In-Memory Joins can be configured as a fallback strategy

Support for SharePoint On-prem with Windows authentication has been added.

Test connection enhancements – the Warning/Error icon now shows a tooltip which provides better information about company connections in error (e.g. loss of authentication token)

In order to improve security, the option to authenticate to webservices using a Webservice API key has now been deprecated in sandbox environments. Only Azure Active Directory should be used.

For incremental source loading to work accurately we need a column that is always:

When either or neither of these requirements are met, it causes inaccuracies.

An improvement was made in the Incremental Load date column logic to ensure the requirements are always met.

Microsoft Dataverse can now be accessed using ZAP’s OData connector as all data in Dataverse is stored within the OData specification.

Dataverse requires Azure Active Directory authentication. Instructions will be available to guide you through the authentication registration.

More information about Dataverse

PKCE has been added as an additional authentication method for REST. More information on PKCE