Data Mesh Resources

Zhamak Dehghani on Data Mesh, Domain-Oriented Data, and Building Data Platforms

In this podcast, Daniel Bryant sat down with Zhamak Dehghani, principal consultant, member of technical advisory board, and portfolio director at ThoughtWorks. Topics discussed included: the motivations for becoming a data-driven organization; the challenges of adapting legacy data platforms and ETL jobs; and how to design and build the next generation of data platforms using ideas from domain-driven design and product thinking, and modern platform principles such as self-service workflows.

Read Article

Data Mesh: Delivering Data-Driven Value at Scale

We're at an inflection point in data, where our data management solutions no longer match the complexity of organizations, the proliferation of data sources, and the scope of our aspirations to get value from data with AI and analytics. In this practical book, author Zhamak Dehghani introduces data mesh, a decentralized sociotechnical paradigm drawn from modern distributed architecture that provides a new approach to sourcing, sharing, accessing, and managing analytical data at scale. Dehghani guides practitioners, architects, technical leaders, and decision makers on their journey from traditional big data architecture to a distributed and multidimensional approach to analytical data management. Data mesh treats data as a product, considers domains as a primary concern, applies platform thinking to create self-serve data infrastructure, and introduces a federated computational model of data governance.

Read Article

Data Mesh Principles and Logical Architecture

Data Mesh Principles and Logical Architecture

Read Article

Data Meshes: Big Data Architecture Becoming Distributed, Declarative and Domain Oriented

Beyond The Data Lake was Director of Emerging Technologies at ThoughtWorks, Zhamak Dehghani's 2017 paper that was a guiding light for Sam Ramji at another point in his career. Listen to how a Data Mesh allows the composition of multi-model data across an organization and beyond.

Read Article

How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh

Many enterprises are investing in their next generation data lake, with the hope of democratizing data at scale to provide business insights and ultimately make automated intelligent decisions. Data platforms based on the data lake architecture have common failure modes that lead to unfulfilled promises at scale. To address these failure modes we need to shift from the centralized paradigm of a lake, or its predecessor data warehouse. We need to shift to a paradigm that draws from modern distributed architecture: considering domains as the first class concern, applying platform thinking to create self-serve data infrastructure, and treating data as a product.

Read Article

Data Mesh: The Four Principles of the Distributed Architecture

A data mesh is a decentralised architecture devised by Zhamak Dehghani, director of Next Tech Incubation, principal consultant at Thoughtworks and a member of its technology Advisory Board.

Read Article

Getting Started with Data Mesh

"To start, it makes more sense to answer “why is data mesh”. Zhamak Dehghani (creator of the data mesh concept) had worked with many large-scale companies on the operational side, helping them build scalable applications and platforms. She looked at the way most companies (try to) use data/organize their analytics organization and came to the conclusion: This is a hot mess express. And I’m going to try to fix it. [our phrasing] Data mesh is her attempt to enable companies to actually be data driven by changing the way they organize their teams and their data architecture. And it is a big change from the status quo. That is a VERY BIG challenge to take on and the size of that challenge is why her first post on data mesh is 6,500+ words. Turns out completely changing the ways company organize is a complicated and dense topic. So, what is a data mesh? It’s a change in approach to 1) use data product thinking - approaching your data as a product instead of a by-product of how you do business; and 2) decentralize data ownership (so data/data quality is no longer owned by the centralized data lake team) and decentralizing data architecture. It isn’t a technology, it’s an approach to being data driven."

Read Article

The Data Lake is dead – Long Live Data Mesh

In this interview I speak with Jon Cooke of Dataception about the emerging new domain of the data mesh. If you and your company are focused. on building data lakes you might want to think again.. Watch this interview to find out more. Our very first in the interview series - Data Lakes are Dead, Long Live Data Mesh.

Watch Video

Data Mesh by Zhamak Dehghani

By: Zhamak Dehghani

This talk will cover an introduction to Data Mesh and the motivations behind it - the failure modes of past paradigms of big data management. Zhamak will compare and contrast data mesh to existing big data management approaches, and introduce the technical components underpinning the architecture

Watch Video

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes Beyond the Data Lake

The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities, a lack of data ownership, and sub-par data availability. At Zalando – europe’s biggest online fashion retailer – we realized that accessibility and availability at scale can only be guaranteed when moving more responsibilities to those who pick up the data and have the respective domain knowledge – the data owners – while keeping only data governance and metadata information central. Such a decentralized and domain focused approach has recently been coined a Data Mesh. The Data Mesh paradigm promotes the concept of Data Products which go beyond sharing of files and towards guarantees of quality and acknowledgement of data ownership. This talk will take you on a journey of how we went from a centralized Data Lake to embrace a distributed Data Mesh architecture backed by Spark and build on Delta Lake, and will outline the ongoing efforts to make creation of data products as simple as applying a template.

Watch Video

Intuit’s Data Mesh Journey – Past, Present, and Future

Tristan Baker and Suresh Raman from Intuit share Intuit's data mesh journey so far and where they are headed in the future. They shared insights into getting buy-in from the org, alignment, global definition management, cross-domain harmonization, governance, and many more topics. They are still in the early days of their data mesh journey but want to share what learnings they have and get feedback from others

Watch Video

Data Mesh at DPG Media: Increasing News Personalization Relevance

Wannes Rosiers, Area Manager for Data (and News Personalization) share how DPG Media transitioned from a non-scalable data organization and architecture to one with the agility necessary to meet customer needs and make data-driven decisions. Recorded May 20, 2021, hosted by Data Mesh Learning (DML), a vendor-independent resource for learning about data mesh.

Watch Video

Data Mesh at HelloFresh – A Work in Progress

The HelloFresh team covered their data mesh journey to-date and what they have learned along the way. Christoph Sawade (Sr. Director Data, Technology) and Hariprasad Natarajan (Lead Data Architect) talked about the history of data at HelloFresh the challenges of data modelling in a data mesh. Mario Konschake (Product Manager Data Foundation) shared an example of how a self-serve data platform is used as an enabler for domain data teams, followed by Surya Emmylinda (Senior Data Product Manager) and Sharif Abdel-Halim (Sr. Data Engineer) who presented an exemplary data product in more detail; Pedro Castillo (Sr. Data Engineer) talked about how to foster a data product mindset. Natalie Hallak (Director Data Management) concluded the presentation by sharing HelloFresh's experience with increasing data literacy.

Watch Video

Data Mesh Conversation with Zhamak Dehghani, Barr Moses, and Lena Hall

Twitter Spaces discussion with Zhamak Dehghani (ThoughtWorks), Barr Moses (Monte Carlo), and Lena Hall (Microsoft) around Data Architecture, Data Mesh, and Data Quality. Listen to learn more about the challenges common in many organizations, pillars, benefits, and vision of the Data Mesh concept, common points of confusion in understanding Data Mesh, data observability, data quality, and many more topics.

Watch Video

Kolibri Games’ Data Mesh Journey and Measuring Data Mesh ROI

António Fitas gives a history of Kolibri Games' journey with data from inception until now as they have begun implementing data mesh. For every year, he talks about the evolution of teams on the data/engineering side as well as the pain points of their setup at that point. Barr and Scott of Monte Carlo discuss how to evaluate whether data mesh is right for you and how to measure the return on investment of data mesh, especially the increase in agility and the increase in the number of decisions you deem "data driven".

Watch Video

Data Lake Strategy via Data Mesh Architecture at JPMorgan Chase

Three very senior leaders at JPMorgan Chase presented about their data mesh journey so far, including moving from on-prem to the cloud, with true honesty about what is working and what is a work-in-progress. Topics included how they are mapping domains to high level products with a number of sub-products and how they are handling a very strict regulatory environment (basically - let's not try to move the really tough stuff to data mesh first).

Watch Video

Anatomy of a Data Product in Data Mesh

Mohammad Syed (Credera) and Pedro Castillo (HelloFresh) joined this meetup to discuss what a data product actually means in a data mesh context. There was a definition section covering the scope of "how big" re a data product, the architectural components of a data product (including a terrible analogy by Scott), and more. Concrete recommendations from the meetup included always try to use a pull method for your data input port, start overly large on the first data product in a domain and decompose as necessary, create a culture of data consumers being able to request changes rather than trying to serve every need upfront, it is okay for your first data product to be a spreadsheet, and more

Watch Video

The Cost of Choosing to Not Version Your Data

Gavin and Paul covered a wide range of topics relative to data versioning, both for data mesh and in general. Per Zhamak, versioning is a necessary capability of data products in data mesh, similar to the DevOps world. However, versioning with data/data products isn't just the code to produce/support the data products but also covers versioning of the data itself.

Watch Video

Data Mesh at CMC Markets: Past, Present and Future

CMC Markets approached data mesh from a slightly different angle from most: they were already organized in silos but were not sharing information well across the organization. Their unique perspective gives them unique insights to some of the challenges of implementing a data mesh. In the presentation, Lorenzo Nicora, Michał Stypik, and Tareq Abedrabbo of the Core Data team at CMC Markets shared the story of the company’s journey so far. They highlighted the architectural, technical, and organizational challenges they had to address, while identifying the approaches they took to shift the organization towards a Data Mesh and to build the required capabilities (e.g. Data Discovery and bridging the gap between cloud/on-premise data systems).

Watch Video

Data Mesh Behind the Walls – Saxo Bank

"Presented by Sheetal Pratik of Saxo Bank and Divya Joshi and Madhu Podila at Thoughtworks. At a high level, speakers the team covered: - Introduction to Saxo Bank - Why Data is important for Saxo Bank? - Data Mesh - The convergence of Distributed Domain Driven Architecture - Partnership with ThoughtWorks - Data Mesh – Behind the walls"

Watch Video

Panel: Data Discovery in Data Mesh

Panel of experts discussing data discovery, especially re data mesh. Moderator: Paco Nathan (Derwen.ai); Panelists: Shinji Kim (Select Star), Sophie Watson (Red Hat), Mark Grover (Stemma), and Shirshanka Das (Acryl Data). Co-hosted with Data Mesh Learning and the Open||Source||Data Podcast. The mad scientist himself, Paco Nathan (bio), moderated a panel discussion on data discovery. Data discovery is one of the hardest problems to solve in data management in general and comes up in many discussions as a major pain point in most data mesh discussions.

Watch Video

Sharing is Caring – Data Mesh on Cisco IOx (AutoZone)

Pete Brown presented on how he built a data mesh for networking data at AutoZone. Pete discussed an approach to implementing a data mesh using a service mesh with a network-embedded control plane. By combining a purpose-designed mesh protocol with the application hosting features of Cisco IOx devices, Pete demonstrated a method of building a resilient mesh which minimizes infrastructure and middleware dependencies.

Watch Video

Retailer AO’s Data Mesh Plans for an Event-Focused Mesh

Jon Vines, Head of Data Engineering and Integrations at AO (large UK-based retailer of electronics/home goods) presented on the company's plans to build out a data mesh. Jon covered a lot of great topics including the blurring of operational and analytical planes/workloads, their evolution towards data sharing on the operational plane through an event mesh, their pre-work on data mesh interoperability, how they are approaching including domains that are not using events when most of their teams will be sharing via events, and much more

Watch Video

Building an Analytics Ecosystem of Global Data Products at Adevinta

"Sandra Real (Product Manager for Analytics Products) and Xavier Gumara Rigol (Senior Engineering Manager, Experimentation & Analytics Solutions) from Adevinta gave an excellent presentation on how they approach treating data as a product. Adevinta moved most of their data ownership from a centralized team to domain teams but a central team still owns some of the core data sets that are used by many domains. "

Watch Video

Data Mesh at Flexport: Driving Buy-in and Social/Org Challenges

"Abhi Sivasailam, data and growth leader at Flexport, presented on their data mesh architecture. “Data Mesh” is often called a socio-technical evolution, and, while much has been said about the technical, this talk will spent just as much on the social; specifically: "

Watch Video

Simple Explanation of Data Mesh

"Scott Hirleman (of Data Mesh Learning) boils data mesh down to some basics including the difference between ""a data mesh"" (a set of read-only analytical-focused data products made of general purpose data) and data mesh as a paradigm. The goal is to make the understanding data mesh a much easier proposition while keeping in mind that people and process are the most important parts of a data mesh implementation. "

Watch Video

Building a Data Team at a Hypergrowth Startup

"Molly Worwerck (Monte Carlo) moderated a great panel on building your data team at a hypergrowth startup/company with Celina Wong (Director Of Analytics and Strategic Insights at TULA Skincare), Clemence Chee (Global Senior Director of Data at HelloFresh) and Trupti Natu (Head of In-Store (Retail) & Card Strategy and Analytics at Afterpay). Topics covered included how to get started as the first data team hire; how to measure and sell the impact of your team/hires; hiring ahead of needs and building real roles, not just heads to service requests; driving buy-in for growing teams; measuring headcount needs; how to hire in a hot market; and many more! It was a great one, check it out! "

Watch Video

Data Mesh: Who Should Data Product Developers

A meetup to discuss the three emerging models of data product development: 1) the embedded data engineer/analytics engineer, 2) an enabling team, or 3) the software engineers. You can see the Slack community discussion here: https://data-mesh-learning.slack.com/...

Watch Video

Data Mesh: Anatomy of a Data Product

Eric Broda led the discussion around the anatomy of a data product in data mesh.

Watch Video

ABN AMRO’s Data Mesh Platform and Architecture 4yr Evolution

If you've wondered how a data platform and decentralized data architecture might look, Rakesh Singh of ABN AMRO - a large bank headquartered in the Netherlands - shared the current state of their data platform and architecture after 4 years of evaluating and evolving their decentralized data approach. It featured a heavy helping of what they learned along the way so hopefully you can avoid the trial and error they had to go through :)

Watch Video

10 Data Mesh Dos and Don’ts

By: Max Schultze

Data Mesh is a Hype and a Buzzword, probably the biggest one in the data industry of late. Everybody talks about it, yet very few know what is at its core or how to apply it in practice. Data Mesh promotes multiple pillars that ultimately request us to make more conscious decisions about how we deal with data in larger organizational contexts. This talk aims to demystify the buzzword and to kickstart your journey by sharing practical advice on positive behaviors you should foster as well as common pitfalls you should try to avoid.

Watch Video

Data Mesh Related Vendor Interview: Masthead Data; Hosted by Nick Heudecker

Nick Heudecker (former Gartner analyst and current Senior Director at Cribl) interviews Yuliia Tkachova, Co-founder and CEO at Masthead Data. Nick and Yuliia covered the problems Masthead Data is trying to solve around data observability in a unique way by not directly querying the data itself, and much more.

Watch Video

Panel: Data Modeling in Data Mesh

By: "Kent Graziano, Veronika Durgin, and Juha Korpela "

Juha Korpela (Chief Product Officer at Ellie Technologies) facilitated this panel on data modeling in data mesh with Veronika Durgin (Head of Data at Saks) and Kent Graziano (The Data Warrior, former Chief Technical Evangelist at Snowflake). Learn why the participants consider Data Vault to be the best choice available right now (the community takes no stand on that :D), why only focusing on the technical aspects of data modeling instead of the business will lead you astray, how to build in iteration and change management to data modeling, and much more!

Watch Video

Data User Experience – An Introduction

Karen Passmore (CEO at Predictive UX) led this discussion with Alice Parker (Data Engineer at DNB) and Wannes Rosiers (Product Manager at Raito). It will help get your head around user experience in data, for consumers and producers.

Watch Video

Topology of Data Product Teams

"This panel features experienced Data Mesh practitioners sharing their insights on operating models for Data Mesh. Topics include data mesh roles/responsibilities, incentives for data product growth, and crucial interactions between various teams in an enterprise data mesh. Panelists and Moderator: * Charlotte Ledoux, President and Co-Founder, Vallai * Amy Raygada, Senior Data, AI & Analytics Product Manager, Swiss Market Place Group * Jean-Georges Perrin, Intelligence Platform Lead, PayPal * [Moderator] Eric Broda, President, Broda Group Software Inc. "

Watch Video

Data Mesh Operating Model – The key to a successful data mesh journey?

"This panel features experienced Data Mesh practitioners sharing their insights on operating models for Data Mesh. Topics include data mesh roles/responsibilities, incentives for data product growth, and crucial interactions between various teams in an enterprise data mesh. Panelists and Moderator: * Sarita Bakst, Head of Data Management Product Line, JPMC * Ammara Gafoor, Principal Business Analyst , Thoughtworks Germany * Jean-Georges Perrin, Intelligence Platform Lead, PayPal * [ Moderator] Eric Broda, President, Broda Group Software Inc. "

Watch Video

What the Heck is a Data Quantum?

Data Mesh Roundtable: What the Heck is a Data Quantum? With the participation of: Yuliia Tkachova, Tom De Wolf, Michael Toland, Paul Cavacas, Samia Rahman, Andrey Goloborodko, Austin Kronz

Watch Video

Two Years of Data Mesh at Roche Diagnostics

By: "Omar Khawaja, Roche Diagnostics "

Roche Diagnostics’ data mesh journey began in early 2021 and today they have five global functions delivering high quality data products that have drastically accelerated the speed and scale of data-driven intelligence–from five years to three months in some cases. Khawaja will share the steps for their data mesh implementation, as well as a specific use case for safety stock raw material inventory forecast that demonstrates how Roche Diagnostics was able to improve stocking, procurement, and overall impacts to customers. We’ll also hear what’s coming next in the company’s data mesh journey.

Watch Video

Federated Computational Governance with Metadata-driven Automation

By: Amber Hilton, Google Cloud & Thinh Ha, Google Cloud Professional Services

Metadata management is a central component for data governance in Life Sciences, for example, when implementing FAIR principles using open standards such as GFF3. In this talk, we show how metadata-driven automation will allow organizations to scale and embed metadata management into daily developer workflows. The talk will outline an events-driven architecture for metadata-driven automation.

Watch Video

Data Mesh in Pharma & LifeScience: Insights from Industry Leaders

By: Ammara Gafoor, Thoughtworks & Antonio Agudo from Roche Diagnostics

This talk will provide an insider view of the different data mesh strategies adopted by some of the top 10 Pharma and LifeScience companies across the world. We will take a deep dive into one specific data mesh strategy hearing directly from the Data Mesh Program Manager at Roche. The talk will touch upon focus areas, main challenges, mitigations and definitive next steps in their respective data mesh journeys.

Watch Video

The RWD Store – Accelerating Evidence Generation with Data Mesh

By: Alexandra Grebe de Barron, Bayer

Bayer runs a Real-World Data Store (RWD), the human- and machine-friendly self-service data product that accelerates real-world evidence generation across the lifelong patient pathway– from pre-diagnosis to treatment. The company has achieved this by embracing data mesh and the data product approach, shifting the mindset internally from application-centric thinking to data-centric thinking with data exposed to the many users and many domains. In this talk de Barron will discuss why data mesh and data products accelerate evidence generation, how they achieved this within Bayer and what were the challenges and learnings.

Watch Video

The Federated Data Governance Journey

By: Mariana Hebborn PhD, Merck Gruppe

In a data mesh implementation, federated data governance paves the way for broad experimentation and innovation in data in ways that were not previously possible. But an organization must carefully consider how to do this safely, and how to bring stakeholders along on the journey. In this talk Hebborn will discuss the importance for an organization to define why to invest in data governance up front, how to inspire stakeholders by sharing expected benefits, and empowering them to contribute by setting clear processes and procedures. Throughout the talk, she’ll refer to how this was successfully managed at Merck Gruppe.

Watch Video

Data Mesh: Delivering Data-Driven Value at Scale

By: Zhamak Dehghani

Now that you have a firm understanding of data mesh as a concept, we recommend you read the O’Reilly book by Zhamak

Read Article

How and why successful data-driven companies are adopting Data Mesh

By: Agile Lab

This post makes some very good observations about the current state of data management and why data mesh is different.

Read Article

What is a Data Mesh — and How Not to Mesh it Up

By: Barr Moses of Monte Carlo

A good overview of data mesh. The figure below is a very interesting competitive differentiator of data mesh. Also, this article does a good job of creating a “is data mesh right for my company” test. Check it out.

Read Article

Data Mesh Principles and Logical Architecture

By: Zhamak Dehghani

Zhamak’s second article on Martin Fowler’s blog where she delves deeper into data mesh. It’s another incredibly well written piece that answers a lot of the lingering questions from her initial post.

Read Article

Data Meshes: Big Data Architecture Becoming Distributed, Declarative and Domain Oriented

By: Zhamak Dehghani & Sam Ramji

Zhamak’s appearance on Sam Ramji’s Open Source Data podcast gives a bit of a different flavor in that it is less interview about data mesh and more of a conversation that touches on many topics. It covers microservices and service mesh’s influence on data mesh and how Zhamak adapted her thinking from the operational plane to apply to the analytics plane.

Listen Now

How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh

By: Zhamak Dehghani

It is a very dense but incredibly well-written and organized overview, offering a solution to the VERY large problem impacting many companies using a data lake. Zhamak prescribes to the saying of “you can’t just point out a flaw, you need to offer a solution” so she did.

Read Article

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes Beyond the Data Lake

By: Zalando and ThoughtWorks

A presentation by Zalando and ThoughtWorks on Zalando’s data mesh implementation. It really crystalizes the way you could implement a data mesh yourself e.g. how to offer self-serve data infrastructure.

Read Article

Data Mesh: The Four Principles of a Distributed Architecture

By: Eleks

After the Getting Started page and these two pieces of content, you should understand quite a bit about data mesh.

Read Article