Optimize model processing time
Models that process quickly deliver fresh data to your reports sooner, so your team spends less time waiting and more time making decisions. Efficient processing also reduces the chance of scheduled processes overlapping or queuing, and frees up resources so that other models and users aren't slowed down.
Schedule fewer processes
- Reduce the frequency of scheduled process configurations.
- Remove process configurations that are no longer needed.
- Use pipeline tags to process only the pipelines that require frequent updates instead of the entire model.
Use on-demand refresh instead of scheduled processing
- Use on-demand refresh to let report users trigger a lightweight refresh of only the data needed for the current report.
- This eliminates the need for frequent scheduled processes.
- See pipeline requirements for on-demand refresh.
Optimize pipeline design
- Configure incremental source loading and incremental warehouse loading so that only new or updated data is processed. Use the Add Only quick incremental type for transaction pipelines where existing rows do not change, and avoid scheduling regular full refreshes. See incremental loading for more detail.
- Use the Create pipeline as Auto setting to let Data Hub choose the optimal materialization for each pipeline, and create views for trivial pipelines.
- Remove unnecessary columns and resolve all validation warnings.
- See The complete guide to modeling in Data Hub for detailed optimization guidance.