Data fragmentation remains a significant challenge, affecting the efficiency and reliability of data analytics in large organizations. Despite advancements in central data governance, most organizations continue to suffer from the proliferation of redundant, poorly organized, and seldom-used data tables. This presentation introduces a new Unified Data Layer (UDL) approach to address these challenges. UDL aims to serve as a flexible, consolidated layer above existing data warehouses, designed to eliminate data silos and encourage reusability. By employing a structural model inspired by Object-Oriented Programming (OOP), the UDL creates a collection of source and processed views, effectively serving as encapsulated units of relevant data and related operations.

Key Takeaways:

• The current data landscape is littered with fragmented, poorly maintained tables that hinder efficient data discovery and analysis.
• UDL offers a modular approach to organizing and unifying data inspired by principles derived from the evolution of programming paradigms, notably OOP.
• Implementing UDL can significantly reduce the number of ad-hoc pipelines and redundant tables, thereby improving data discoverability and reusability.
• With UDL, multiple teams can contribute data related to a single entity, which can be joined from various sources to present a comprehensive view, leading to more effective decision-making.
• By implementing UDL, organizations can overcome some of the most pressing challenges in data management, paving the way for a more organized, efficient, and effective utilization of data assets.

Speaker: Mitesh Mangaonkar 

Mitesh is an engineering leader pioneering the use of generative AI to transform data platforms. As a tech lead at Airbnb, he builds cutting-edge data pipelines leveraging big technologies and modern data stacks to power trust and safety products. Previously, at AWS, Mitesh helped Fortune 500 companies migrate their data warehouses to the cloud and engineered highly scalable, resilient systems. An innovator at heart, he combines deep data engineering expertise with a passion for AI to create the next generation of data products. Mitesh is an influential voice shaping the future of data engineering and governance.