The evolution of software engineering has embraced microservices and modular design, enabling resilience and adaptability in complex systems. However, data engineering remains rooted in monolithic architectures and outdated practices. This talk draws from decades of experience in data and software engineering, advocating for the application of modern software principles to data systems. By embracing modularity, data contracts, and tools like dbt, data professionals can address challenges such as siloed systems, centralized ownership, and inflexible processes. This session will provide practical guidance for shifting toward an anti-monolith architecture and fostering innovation in data engineering.

Key focus areas:

• Data Mesh: An understanding of this socio-technical approach to decentralizing data architecture and enabling interoperability.
• Modularity in Data Engineering: Breaking down monolithic data systems into independent components integrated via data contracts.
• Evolutionary Architecture: Leveraging microservice patterns for low-risk upgrades and adaptability in data systems.
• Adopting Software Principles: Implementing CI/CD, automated testing, and observability in data engineering.
• Tools and Technologies: Utilizing dbt, automated quality checks, and cross-project references to modernize workflows.
• Practical Next Steps: Incremental actions for embracing anti-monolith design and modernizing data practices.

Speaker: Elliott Cordo 

Elliott is an expert in cloud-native data engineering, strategy, and architecture, with a passion for helping organizations drive value from data. He has more than two decades of experience implementing cutting-edge, data-driven cloud-native applications. He has a passion for helping organizations understand the true potential of their data by working as a leader, architect, and builder. Elliott has built dozens of cloud-native data platforms on AWS, ranging from data warehouses and data lakes, AI, and machine learning infrastructure to real-time activation platforms in companies ranging from small startups to large enterprises. He is the founder of Data Futures, a consultancy focused on cloud-native data modernization, and an AWS Data Hero. https://aws.amazon.com/developer/community/heroes/elliott-cordo/