README.md

Works With Determined

This repository highlights tools in the ML ecosystem that work well with Determined. It contains working examples of full end to end machine learning workflows that are enabled with the right sets of tools.

Data Tools

• Pachyderm
  • An example Pachyderm integration can be found in the Pachyderm-Seldon example
• DVC
  • An example of using DVC to version data for Determined can be found in the DVC example
• Delta Lake
  • An example of reading data from a Delta table to train a model in Determined can be found in the spark example

Serving Tools

• Algorithmia
  • An example of using Determined with Algorithmia to train and deploy an Object Detection model can be found in the Algorithmia example
• Seldon Core
  • An example of using Seldon as a part of an end to end platform can be found in the Pachyderm-Seldon example
  • An example demonstrating automatic Seldon serving of models trained in Determined can be found in the argo workflow example
• Spark
  • An example that uses Spark to perform batch inference can be found in the spark example

Workflow Tools

• Argo
  • An example of using Determined within an Argo workflow to train a model then automatically deploy it with Seldon Core can be found in the Kubeflow pipelines example
• Airflow
  • An example of using Determined within an Airflow workflow to train a model then deploy it into Kubernetes with Seldon core can be found in the airflow example
• Pachyderm
  • An E2E example of using Determined within an Pachyderm pipelines to train a model using pachyderm repo for data management and deploy using KServe Inference.

Platforms

• Kubeflow
  • The Kubeflow pipelines example shows how to build a Kubeflow Pipeline with the Kubeflow Pipeline DSL that trains a model with Determined, saves improvements to the Determined model registry, then deploys that model with Seldon Core.

ML Libraries

• RAPIDS
  • The NVIDIA RAPIDS example shows how to integrate RAPIDS into model training on Determined, in order to perform GPU-accelerated data loading and preprocessing.

Observability Tools

• Prometheus
  • prometheus.yml is a Prometheus configuration that works with the Prometheus endpoint that Determined surfaces to allow external observability tools to work with Determined
• Grafana
  • Determined Hardware Dashboard is a pre-configured Grafana dashboard that contains queries and panels that integrate with a Prometheus endpoint to visualize cluster usage metrics

Cluster Managing Tool

• HPCM