mlops.md

This page covers MLOps and DataOps/Eng topics. I started this in early 2020. Also see dataeng

General Articles, Blogs, and Talks

• Hidden Tech Debt in Machine Learning Systems - 2015 paper that laid the foundations
• InnoQ MLOps Site - this looks dormant but provides the state of things and problem definition in late 2019 early 2020

Principles & Meta

• What are Model Governance and Model Operations (OReilly, June 2019)
• DataOps Manifesto
• The Rise of the Term “MLOps:” Properly Operationalized Machine Learning is the New Holy Grail
• MLOps: ML Engineering Best Practices from the Trenches - 2019
• Bristech 2019 Luke Marsdon - The future of MLOps - 2019
• Reimagining DevOps for ML by Elle O'Brien, Iterative.ai - 2020
• Machine learning deserves its own flavor of Continuous Delivery - April 2020
• MLOps — Is it a Buzzword??? Part -1 - June 2021, Walmart Labs

Practices

• Building an End-to-End MLOps Pipeline with Open-Source Tools - Nov 2023
• This is how you set up an MLOps platform on AWS EKS with Kubeflow and MLflow
• Productionizing Machine Learning Models with MLOps
• Deploying ML Models in Distributed Real-time Data Streaming Applications
• Architecting a Machine Learning Pipeline How to build scalable Machine Learning systems — Part 2/2
• Being a Data Scientist does not make you a Software Engineer!
• Deploying Python ML Models with Flask, Docker and Kubernetes
• Deploying and Versioning Data Piplines at Scale
• CI/CD + ML == MLOps - The Way To Speed Bringing Machine Learning To Production - David Aronchick - this as well!
• Data Pipelines @ Samsara - March 2021
• Continuous Delivery For Machine Learning: Patterns And Pains - Emily Gorcenski - great talk, I saw this in person in January 2020.

Maturity Modesl

• Microsoft MLOps Maturity Model

Relevant Research & Serious Articles

• Hidden Tech Debt in Machine Learning Systems
• Continuous Delivery for Machine Learning
• Coding habits for data scientists
• Continuous Delivery for Machine Learning: Automating the end-to-end lifecycle of Machine Learning application
• What Is MLOps And Why Your Team Should Implement It

Conferences & Workshops

• https://www.mlopsnyc.com/agenda-sessions - see https://www.youtube.com/channel/UChmi6ZzsZd9doYYVut1ppUg
• https://www.usenix.org/sites/default/files/opml19_full_proceedings.pdf
• https://github.com/thoughtworksInc/CD4ML-Scenarios

Tooling

• Deploying Python ML Models with Flask, Docker and Kubernetes
• MLOps, Kubeflow, and Tekton - Simon Kaegi, IBM
• End-to-end Machine Learning Platforms Compared

Dagster

• See Dagster

Metaflow (Netflix)

• Metaflow
• source
• Open-Sourcing Metaflow, a Human-Centric Framework for Data Science

Mlfow (DataBricks)

• Simplifying Model Management with MLflow
• mlfow - an open source platform for managing the end-to-end machine learning lifecycle. It tackles three primary functions: 1) tracking experiments to record and compare parameters and results 2) packaging ML code in a reusable, reproducible form in order to share with other data scientists or transfer to production 3) managing and deploying models from a variety of ML libraries to a variety of model serving and inference platforms (MLflow Models). By Databricks

Feature Stores

• Feast
• Hopsworks

Kubeflow

• Accelerating Machine Learning App Development with Kubeflow Pipelines (Cloud Next '19
• Kubeflow: Simplified, Extended and Operationalized

General K8s

• Seldon - Seldon Core an open source platform for deploying machine learning models on a Kubernetes cluster.
• Argo CD
• Flyte - a structured programming and distributed processing platform created at Lyft that enables highly concurrent, scalable and maintainable workflows for machine learning and data processing.
• [

Mleap

• https://medium.com/rv-data/mleap-providing-near-real-time-data-science-with-apache-spark-c34e7df093ca
• https://github.com/combust/mleap

Spark

• Best Practices for Building and Deploying Data Pipelines in Apache Spark - Vicky Avison
• Waimak - an open-source framework that makes it easier to create complex data flows in Apache Spark

Data Versioning & Validation

• DVC - is built to make ML models shareable and reproducible. It is designed to handle large files, data sets, machine learning models, and metrics as well as code.

DBT

• DBT
• Taming the Dependency Hell with dbt - March 2021

Great Expectations

• Great Expectations - helps teams save time and promote analytic integrity by offering a unique approach to automated testing: pipeline tests. Pipeline tests are applied to data (instead of code) and at batch time (instead of compile or deploy time). Pipeline tests are like unit tests for datasets: they help you guard against upstream data changes and monitor data quality and https://greatexpectations.io/
• Using GitHub Actions for MLOps & Data Science
• Keeping your data pipelines healthy with the Great Expectations GitHub Action
• Great Expectations: Validating datasets in machine learning pipelines
• Database Testing with Great Expectations - June 2020

Other Open Source Tools

• Apache Camel
• Airbyte
• Benthos (now RedPanda Connect
• Kedro - an open source development workflow tool that helps structure reproducible, scaleable, deployable, robust and versioned data pipelines (by Quantum Black Labs])
• H20 AutoML
• Meltano
• Firebolt
• Pachyderm and Deploy on a Cloud via K8s
• PrefectCore
• PipelineWise
• Singer

Lineage and Metadata Management

• DataHub
• OpenLineage
• Apache Atlas
• Marquez
• Tokern

Observability

• Databand

Vendors

• Dot Science
• Data Kitche
• Hydrosphere
• StreamSets - see https://github.com/streamsets

Cloud Offerings

• Qwak

Google

• Architecture for MLOps using TFX, Kubeflow Pipelines, and Cloud Build
• Google

Azure

• https://azure.microsoft.com/en-us/services/machine-learning/mlops/
• https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment