Case Study – Data Mesh – Federated Query and Access Control Over Data Products

As Zhamak Dehghani defines it in her seminal book “Data Mesh: Delivering Data-Driven Value at Scale,” data mesh is based on four core principles:

Decentralized domain ownership

Data as a product

Self-serve data platform

Federated computational governance

She writes, “The data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.” These principles ensure an improved level of reliability and ease of use for consumers, while still maintaining high data protection standards.

In this talk, I will outline an approach to building an enterprise data mesh to honor these principles. Our data mesh spans a wide variety of data products, including on-prem and public cloud and SQL, noSQL, and API-only access. We support federated queries across these data products using Presto/Trino and Spark SQL. Fine-grained role-, tag-, and attribute-based access control are provided using extensions of Apache Atlas and Ranger, fully automated after an initial metadata specification. Self-service discovery and access control are provided on a column/attribute level.

Our data mesh vision is not yet fully realized. I’ll share the progress we’ve made toward this vision and the lessons we’ve learned along the way.

 Speaker: Barbara Eckman

Distinguished Architect, Global Technology Organization, Comcast

Barbara Eckman is a Distinguished Architect in the Global Technology Organization at Comcast. She is Lead Architect for Enterprise Metadata, Lineage, and Access Control, with a particular focus on streaming, transforming, governing, analyzing and protecting both Big Data and distributed relational data sources. Her prior experience includes scientific data and model integration at the Human Genome Project, Merck, GlaxoSmithKline, and IBM, where she served on the peer-elected IBM Academy of Technology.