Talks

This repository is home to presentations and demo projects created by the Dagster team.

If you have any questions, you are welcome to contact us on Slack, or through GitHub discussions.

Table of Contents

• Deep Dive - Dagster Running Dagster: Compass
• Best Practices for LLM Development
• Small Data - Composable Data Workflows: Building Pipelines That Just Work
• All Things Open - Enabling community education
• Deep Dive - Dagster Running Dagster: Streaming
• Deep Dive - dltHub
• Deep Dive - Breaking Biology's Data Wall
• Deep Dive - Cooking with gas: Building a data platform at US Foods
• Deep Dive - Building Breakthrough AI Applications with Not Diamond
• Deep Dive - Shifting Left and Moving Forward
• Deep Dive - Orchestrating AI Agents
• Deep Dive - The Evolution of Data Catalogs
• Deep Dive - Dagster Modal Demo
• Building a True Data Platform: Beyond the Modern Data Stack
• Dagster, SDF, & the Evolution of the Data Platform
• Deep Dive - Data Quality
• Motherduck, Dagster, Evidence Demo
• Deep Dive - Enable Data Mesh
• Deep Dive - Thinking in Partitions
• Deep Dive - Resources & Configurations
• Deep Dive - Automations

Deep Dive - Dagster Running Dagster: Compass

🔗 Resources: Slides • Video Learn how Dagster Labs uses Dagster to power its own internal data platform with Compass.

Best Practices for LLM Development

📅 Date: January 27, 2026 👥 Authors: Alex Noonan, Dennis Hume, Colton Padden 🔗 Resources: Slides Learn best practices for integrating LLMs into Dagster pipelines, covering the development lifecycle, common pitfalls to avoid, and how to leverage the official Claude plugin for AI-powered data orchestration.

Small Data - Composable Data Workflows: Building Pipelines That Just Work

📅 Date: November 4, 2025 👥 Authors: Dennis Hume 🔗 Resources: Slides • Source Code Learn how to build a zero-cost GitHub issue deduplication system using Dagster and DuckDB, covering data ingestion, embeddings generation, and composable workflow automation.

All Things Open - Enabling community education

📅 Date: October 14, 2025 👥 Authors: Colton Padden 🔗 Resources: Slides See how the team at Dagster creates educational content for the community with a little help from AI.

Deep Dive - Dagster Running Dagster: Streaming

📅 Date: August 26, 2025 👥 Authors: Nicholas Roach 🔗 Resources: Slides Learn how Dagster Labs uses Dagster to power its own internal data platform, including implementing streaming event ingestion for Dagster+ and testing experimental features in production.

Deep Dive - dltHub

📅 Date: July 8, 2025 👥 Authors: Alena Astrakhantseva, Aashish Nair, Colton Padden, Alex Noonan 🔗 Resources: Slides Rapidly developing data pipelines with dltHub and Dagster

Deep Dive - Breaking Biology's Data Wall

📅 Date: July 1, 2025 👥 Authors: Keith Kam 🔗 Resources: Slides Learn how Basecamp Research scaled a multi-disciplinary data platform to power BioAI foundation models and overcome the unique challenges of managing biological data at scale.

Deep Dive - Cooking with gas: Building a data platform at US Foods

📅 Date: April 1, 2025 👥 Authors: Lee Littlejohn, Alex Noonan 🔗 Resources: Slides Discover how US Foods transformed their fragmented data infrastructure into a scalable, organized data platform using Dagster, Snowflake, and AWS with infrastructure as code, achieving improved visibility, faster onboarding, and team self-service.

Deep Dive - Building Breakthrough AI Applications with Not Diamond

📅 Date: February 11, 2025 👥 Authors: Alejandro Companioni, Tomas Kofman, Colton Padden 🔗 Resources: Slides Learn how to use Not Diamond's intelligent LLM routing with Dagster to automatically select the best model for each task, optimizing for quality, cost, and latency in your AI pipelines.

Deep Dive - Shifting Left and Moving Forward

📅 Date: January 14, 2025 👥 Authors: Alex Noonan, Colton Padden, Jacob Matson 🔗 Resources: Slides • Source Code Learn how to build modern data pipelines with Dagster and MotherDuck through a hands-on Bluesky data platform demo that demonstrates shifting left in data engineering.

Deep Dive - Orchestrating AI Agents

📅 Date: October 31, 2024 👥 Authors: Olivier Dupuis, Izzy Miller, Colton Padden 🔗 Resources: Slides • Video Learn how to build and orchestrate AI agent prototypes within a data platform using LangChain, Hex, and Dagster to enrich unstructured climate narratives data.

Deep Dive - The Evolution of Data Catalogs

📅 Date: October 15, 2024 👥 Authors: Alex Noonan 🔗 Resources: Slides • Video Learn how data catalogs have evolved to enable data discovery through metadata collection and indexing, and explore Dagster's approach to addressing the limitations of traditional catalog solutions.

Deep Dive - Dagster Modal Demo

📅 Date: September 24, 2024 👥 Authors: Charles Frye, Colton Padden 🔗 Resources: Slides • Source Code • Video Learn how to orchestrate scalable ML workloads by combining Dagster's state management and observability with Modal's auto-scaling infrastructure and GPU provisioning.

Building a True Data Platform: Beyond the Modern Data Stack

📅 Date: September 6, 2024 👥 Authors: Pedram Navid 🔗 Resources: Slides • Video Explore the shortcomings of the Modern Data Stack and discover how to build a unified data platform using Dagster that delivers on the promises of scalability, observability, and maintainability.

Dagster, SDF, & the Evolution of the Data Platform

📅 Date: August 22, 2024 👥 Authors: Lukas Schulte, Pedram Navid 🔗 Resources: Slides • Video Learn how Dagster and SDF combine to bridge the gap between orchestration and transformation, enabling local SQL development, enhanced data quality, and seamless integration between data platforms.

Deep Dive - Data Quality

📅 Date: August 6, 2024 👥 Authors: Colton Padden 🔗 Resources: Slides • Video Learn how to build reliable data platforms using Dagster's data quality features and explore the six key dimensions of data quality: timeliness, completeness, accuracy, validity, consistency, and uniqueness.

Motherduck, Dagster, Evidence Demo

📅 Date: April 18, 2024 👥 Authors: Colton Padden, Alex Monahan 🔗 Resources: Slides Learn how to seamlessly transition data pipelines from local development to production using MotherDuck's hybrid compute architecture with Dagster orchestration, dbt transformations, and Evidence dashboards.

Deep Dive - Enable Data Mesh

📅 Date: April 2, 2024 👥 Authors: Tim Castillo 🔗 Resources: Slides • Source Code • Video Learn how to enable a data mesh paradigm with Dagster by implementing the four core principles of domain-driven ownership, data-as-a-product, federated governance, and self-serve platforms.

Deep Dive - Thinking in Partitions

📅 Date: March 5, 2024 👥 Authors: Tim Castillo 🔗 Resources: Slides • Video Learn how to use Dagster partitions to break the linear relationship between growing data volumes and pipeline resource consumption.

Deep Dive - Resources & Configurations

📅 Date: February 20, 2024 👥 Authors: Colton Padden 🔗 Resources: Slides • Video Learn how to use Dagster's configurations and resources to build reusable, flexible data pipelines that can be tested locally and deployed to production with confidence.

Deep Dive - Automations

📅 Date: February 13, 2024 👥 Authors: Pedram Navid 🔗 Resources: Slides • Video Learn how to automate your Dagster data pipelines using schedules, sensors, and auto-materialize policies to eliminate manual execution and simplify complex workflows.

---