Element / Retail - EU PDS Dashboard Acceleration
EU PDS Dashboard Acceleration
Problem :
The EU PDS dashboard consumed 1.28B records daily from eFundamentals, with pipeline execution taking 6+ hours and incurring high Azure compute costs.
Solution :
Spark partitioning, delta table compression, and selective data overwrite logic were applied using PySpark and Databricks. The pipeline was optimized for performance and deployed via Azure Data Factory.
Outcome :
Execution time was reduced from 6 hours to 2.5 hours. Azure cloud billing dropped by 50%, significantly improving operational efficiency and cost control.