Entrada

The Challenge

A leading Sales Automation software platform was facing challenges in maximizing the performance of its existing Databricks pipelines. The client’s goal was to improve data processing efficiency and evaluate the potential for migrating its EMR streaming jobs. This is where Entrada stepped in to help optimize data infrastructure and support growth objectives.

The Solution

Entrada collaborated closely with the client to deliver a comprehensive assessment of their current data infrastructure and provide actionable recommendations for enhancing performance and return on investment (ROI). The engagement focused on the following key areas:

Performance Optimization of Databricks Pipelines: Entrada utilized the latest capabilities of Databricks, including Liquid Clustering, Z-Ordering, and advanced partitioning techniques, to significantly improve the performance of lakehouse pipelines. The enhancements were particularly focused on jobs responsible for managing over 300 data pipelines, ensuring a more efficient and responsive data processing environment.
Migration to Unity Catalog: As a value-added service, Entrada migrated the client’s data pipelines to Unity Catalog. This strategic move not only improved data governance and security, but also laid the foundation for future AI and machine learning initiatives, enhancing the client’s ability to innovate and scale.
Optimization for DLT Compliance: To further streamline operations, Entrada optimized pipelines to be compliant with Delta Live Tables (DLT). This adjustment reduced operational complexities and ensured a more robust and reliable data processing framework.

The Results

Through strategic collaboration with Entrada, the client successfully enhanced the performance and cost-efficiency of its data infrastructure. The optimizations not only delivered immediate value in terms of cost savings and performance improvements, but also positioned the client for future growth, enabling the company to better leverage its data assets for innovation and competitive advantage:

72% Cost Savings: The largest data pipeline saw a 72% reduction in run costs, translating into significant financial savings.
82% Reduction in Initial Load Time: The initial load time for data processing was reduced by more than 82%, accelerating data availability and enhancing operational efficiency.
50% Reduction in Run Time: The runtime for the largest tables was cut by over 50%, leading to faster data processing and improved overall system performance.

About Entrada
Entrada is a Databricks-focused consulting and implementation partner backed by Databricks Ventures. Entrada harnesses the power of Databricks to help customers accelerate their AI + data initiatives. Our expertise in AI/ML, Databricks, and analytics is centered around industry-centric solutions. Our mission is to simplify complex data + AI challenges and support end-to-end transformations, delivering future-ready solutions fast.