Pre-requisites:
Objective
Scale embedding generation by moving from local in-memory sequential processing to a distributed setup capable of handling large-scale Protein Language Models (PLMs) and extensive protein datasets.
To handle larger-scale compute and data effectively within our Dagster pipelines, we need to transition our embedding generation to a distributed architecture.
🏗️ Architecture for Scaled Compute
We will integrate Ray or Apache Spark with Dagster to distribute the embedding generation across multiple nodes (or GPUs in our Kubernetes cluster), rather than processing sequentially on a single machine.
✅ Implementation Checklist
Pre-requisites:
Objective
Scale embedding generation by moving from local in-memory sequential processing to a distributed setup capable of handling large-scale Protein Language Models (PLMs) and extensive protein datasets.
To handle larger-scale compute and data effectively within our Dagster pipelines, we need to transition our embedding generation to a distributed architecture.
🏗️ Architecture for Scaled Compute
We will integrate Ray or Apache Spark with Dagster to distribute the embedding generation across multiple nodes (or GPUs in our Kubernetes cluster), rather than processing sequentially on a single machine.
✅ Implementation Checklist
generate_embeddingsfunction to use a distributed computing framework (e.g., Ray).