As Zhamak Dehghani defines it in her seminal book “Data Mesh: Delivering Data-Driven Value at Scale,” Data mesh is based on four core principles:

• Decentralized domain ownership
• Data as a product
• Self-serve data platform
• Federated computational governance

She writes, “The data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.”  

I will outline an approach to building an enterprise data mesh that seeks to honor these principles. Our data mesh spans a wide variety of data products, including on-prem & public cloud and SQL, NoSQL, and API-only access. We support federated queries across these data products using Presto/Trino and Spark SQL. Fine-grained Role-, Tag-, and Attribute-based Access Control are provided using extensions of Apache Atlas and Ranger, fully automated after an initial metadata specification. Self-service discovery and access control are provided on a column/attribute level.

Our data mesh vision is not yet fully realized. I’ll share the progress we've made toward this vision and the lessons we've learned along the way.

Speaker: Barbara Eckman

Distinguished Architect, Comcast

Barbara Eckman is a Distinguished Architect in Customer Experience Technologies at Comcast. She is Lead Architect for Enterprise Metadata, Lineage, and Access Control, with a particular focus on streaming, transforming, governing, analyzing and protecting Big Data. Her prior experience includes scientific data and model integration at the Human Genome Project, Merck, GlaxoSmithKline, and IBM, where she served on the peer-elected IBM Academy of Technology. She is an active contributor to the Apache Ranger open source codebase.