Element / Retail - EU PDS Dashboard Performance Optimization
EU PDS Dashboard Performance Optimization
Problem :
The EU PDS dashboard consumed 1.28 billion records daily from eFundamentals, leading to long pipeline run times (6 hours) and high Azure compute costs. This slowed reporting and escalated operational costs.
Solution :
Pipeline optimization was performed using Spark-native techniques. Data was written in compressed delta tables with partitioning. Only recent data was replaced, and the entire process was streamlined using efficient cluster configurations.
Outcome :
Pipeline run time dropped from 6 hours to 2.5 hours. Azure costs were reduced by 50%. The dashboard’s refresh cycle improved drastically, delivering timely insights at lower overhead.