-
Notifications
You must be signed in to change notification settings - Fork 4
Ai provider #179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Ai provider #179
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -41,11 +41,14 @@ npm install @mediapipe/tasks-text @mediapipe/tasks-vision @mediapipe/tasks-audio | |
| ### 1. Basic Setup | ||
|
|
||
| ```typescript | ||
| import { register_HFT_InlineJobFns, register_TFMP_InlineJobFns } from "@workglow/ai-provider"; | ||
| import { | ||
| HuggingFaceTransformersProvider, | ||
| TensorFlowMediaPipeProvider, | ||
| } from "@workglow/ai-provider"; | ||
|
|
||
| // Register AI providers | ||
| await register_HFT_InlineJobFns(); | ||
| await register_TFMP_InlineJobFns(); | ||
| await new HuggingFaceTransformersProvider().register({ mode: "inline" }); | ||
| await new TensorFlowMediaPipeProvider().register({ mode: "inline" }); | ||
|
Comment on lines
48
to
+51
|
||
| ``` | ||
|
|
||
| ### 2. Using AI Tasks in Workflows | ||
|
|
@@ -234,37 +237,42 @@ For better performance, especially in browser environments, run AI inference in | |
| #### Main Thread Setup | ||
|
|
||
| ```typescript | ||
| import { register_HFT_ClientJobFns, register_TFMP_ClientJobFns } from "@workglow/ai-provider"; | ||
| import { | ||
| HuggingFaceTransformersProvider, | ||
| TensorFlowMediaPipeProvider, | ||
| } from "@workglow/ai-provider"; | ||
|
|
||
| // Register HuggingFace Transformers with worker | ||
| register_HFT_ClientJobFns( | ||
| new Worker(new URL("./hft-worker.ts", import.meta.url), { type: "module" }) | ||
| ); | ||
| await new HuggingFaceTransformersProvider().register({ | ||
| mode: "worker", | ||
| worker: new Worker(new URL("./hft-worker.ts", import.meta.url), { type: "module" }), | ||
| }); | ||
|
|
||
| // Register MediaPipe with worker | ||
| register_TFMP_ClientJobFns( | ||
| new Worker(new URL("./tfmp-worker.ts", import.meta.url), { type: "module" }) | ||
| ); | ||
| await new TensorFlowMediaPipeProvider().register({ | ||
| mode: "worker", | ||
| worker: new Worker(new URL("./tfmp-worker.ts", import.meta.url), { type: "module" }), | ||
| }); | ||
| ``` | ||
|
|
||
| #### Worker Setup Files | ||
|
|
||
| **hft-worker.ts:** | ||
|
|
||
| ```typescript | ||
| import { register_HFT_WorkerJobFns } from "@workglow/ai-provider"; | ||
| import { HFT_WORKER_JOBRUN_REGISTER } from "@workglow/ai-provider"; | ||
|
|
||
| // Register HuggingFace Transformers worker functions | ||
| register_HFT_WorkerJobFns(); | ||
| HFT_WORKER_JOBRUN_REGISTER(); | ||
| ``` | ||
|
|
||
| **tfmp-worker.ts:** | ||
|
|
||
| ```typescript | ||
| import { register_TFMP_WorkerJobFns } from "@workglow/ai-provider"; | ||
| import { TFMP_WORKER_JOBRUN_REGISTER } from "@workglow/ai-provider"; | ||
|
|
||
| // Register MediaPipe worker functions | ||
| register_TFMP_WorkerJobFns(); | ||
| TFMP_WORKER_JOBRUN_REGISTER(); | ||
| ``` | ||
|
|
||
| ### Model Management | ||
|
|
@@ -286,30 +294,20 @@ await downloadTask.execute(); | |
| ### Custom Job Queue Configuration | ||
|
|
||
| ```typescript | ||
| import { | ||
| JobQueueClient, | ||
| JobQueueServer, | ||
| ConcurrencyLimiter, | ||
| DelayLimiter, | ||
| } from "@workglow/job-queue"; | ||
| import { InMemoryQueueStorage } from "@workglow/storage"; | ||
| import { register_HFT_InlineJobFns, HF_TRANSFORMERS_ONNX } from "@workglow/ai-provider"; | ||
|
|
||
| // Configure queue with custom limits | ||
| const customQueue = new JobQueueServer(HF_TRANSFORMERS_ONNX, AiJob, { | ||
| storage: new InMemoryQueueStorage(HF_TRANSFORMERS_ONNX), | ||
| queueName: HF_TRANSFORMERS_ONNX, | ||
| limiter: new ConcurrencyLimiter(2, 1000), // 2 concurrent jobs, 1000ms timeout | ||
| }); | ||
| import { HuggingFaceTransformersProvider } from "@workglow/ai-provider"; | ||
|
|
||
| const client = new JobQueueClient({ | ||
| storage: new InMemoryQueueStorage(HF_TRANSFORMERS_ONNX), | ||
| queueName: HF_TRANSFORMERS_ONNX, | ||
| // Register with custom queue concurrency (provider creates queue with concurrency: 2) | ||
| await new HuggingFaceTransformersProvider().register({ | ||
| mode: "inline", | ||
| queue: { concurrency: 2 }, | ||
| }); | ||
|
Comment on lines
+299
to
303
|
||
|
|
||
| client.attach(customQueue); | ||
| // Register AI providers | ||
| await register_HFT_InlineJobFns(client); | ||
| // Or skip auto-creation and use your own queue: | ||
| await new HuggingFaceTransformersProvider().register({ | ||
| mode: "inline", | ||
| queue: { autoCreate: false }, | ||
| }); | ||
| // Then register your custom queue with getTaskQueueRegistry().registerQueue(...) | ||
| ``` | ||
|
|
||
| ### Error Handling | ||
|
|
@@ -352,13 +350,13 @@ await task.execute(); | |
| ## Complete Working Example | ||
|
|
||
| ```typescript | ||
| import { HF_TRANSFORMERS_ONNX, register_HFT_InlineJobFns } from "@workglow/ai-provider"; | ||
| import { TextGenerationTask, TextEmbeddingTask, AiJob } from "@workglow/ai"; | ||
| import { Workflow, getTaskQueueRegistry } from "@workglow/task-graph"; | ||
| import { HuggingFaceTransformersProvider } from "@workglow/ai-provider"; | ||
| import { TextGenerationTask, TextEmbeddingTask } from "@workglow/ai"; | ||
| import { Workflow } from "@workglow/task-graph"; | ||
|
|
||
| async function main() { | ||
| // 1. Register the AI provider | ||
| await register_HFT_InlineJobFns(); | ||
| await new HuggingFaceTransformersProvider().register({ mode: "inline" }); | ||
|
|
||
|
Comment on lines
357
to
360
|
||
| // 2. Create and run workflow | ||
| const workflow = new Workflow(); | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2025 Steven Roussey <sroussey@gmail.com> | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { globalServiceRegistry, parentPort, WORKER_SERVER } from "@workglow/util"; | ||
| import { HFT_TASKS } from "./common/HFT_JobRunFns"; | ||
| import { HuggingFaceTransformersProvider } from "./HuggingFaceTransformersProvider"; | ||
|
|
||
| export function HFT_WORKER_JOBRUN_REGISTER() { | ||
| const workerServer = globalServiceRegistry.get(WORKER_SERVER); | ||
| new HuggingFaceTransformersProvider(HFT_TASKS).registerOnWorkerServer(workerServer); | ||
| parentPort.postMessage({ type: "ready" }); | ||
| console.log("HFT_WORKER_JOBRUN registered"); | ||
| } |
79 changes: 79 additions & 0 deletions
79
packages/ai-provider/src/hf-transformers/HuggingFaceTransformersProvider.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| /** | ||
| * @license | ||
| * Copyright 2025 Steven Roussey <sroussey@gmail.com> | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| import { AiProvider, type AiProviderRegisterOptions, type AiProviderRunFn } from "@workglow/ai"; | ||
| import { HF_TRANSFORMERS_ONNX } from "./common/HFT_Constants"; | ||
| import type { HfTransformersOnnxModelConfig } from "./common/HFT_ModelSchema"; | ||
|
|
||
| /** | ||
| * AI provider for HuggingFace Transformers ONNX models. | ||
| * | ||
| * Supports text, vision, and multimodal tasks via the @sroussey/transformers library. | ||
| * | ||
| * Task run functions are injected via the constructor so that the heavy | ||
| * `@sroussey/transformers` library is only imported where actually needed | ||
| * (inline mode, worker server), not on the main thread in worker mode. | ||
| * | ||
| * @example | ||
| * ```typescript | ||
| * // Worker mode (main thread) -- lightweight, no heavy imports: | ||
| * await new HuggingFaceTransformersProvider().register({ | ||
| * mode: "worker", | ||
| * worker: new Worker(new URL("./worker_hft.ts", import.meta.url), { type: "module" }), | ||
| * }); | ||
| * | ||
| * // Inline mode -- caller provides the tasks: | ||
| * import { HFT_TASKS } from "@workglow/ai-provider"; | ||
| * await new HuggingFaceTransformersProvider(HFT_TASKS).register({ mode: "inline" }); | ||
| * | ||
| * // Worker side -- caller provides the tasks: | ||
| * import { HFT_TASKS } from "@workglow/ai-provider"; | ||
| * new HuggingFaceTransformersProvider(HFT_TASKS).registerOnWorkerServer(workerServer); | ||
| * ``` | ||
| */ | ||
| export class HuggingFaceTransformersProvider extends AiProvider<HfTransformersOnnxModelConfig> { | ||
| readonly name = HF_TRANSFORMERS_ONNX; | ||
|
|
||
| readonly taskTypes = [ | ||
| "DownloadModelTask", | ||
| "UnloadModelTask", | ||
| "TextEmbeddingTask", | ||
| "TextGenerationTask", | ||
| "TextQuestionAnswerTask", | ||
| "TextLanguageDetectionTask", | ||
| "TextClassificationTask", | ||
| "TextFillMaskTask", | ||
| "TextNamedEntityRecognitionTask", | ||
| "TextRewriterTask", | ||
| "TextSummaryTask", | ||
| "TextTranslationTask", | ||
| "ImageSegmentationTask", | ||
| "ImageToTextTask", | ||
| "BackgroundRemovalTask", | ||
| "ImageEmbeddingTask", | ||
| "ImageClassificationTask", | ||
| "ObjectDetectionTask", | ||
| ] as const; | ||
|
|
||
| constructor(tasks?: Record<string, AiProviderRunFn<any, any, HfTransformersOnnxModelConfig>>) { | ||
| super(tasks); | ||
| } | ||
|
|
||
| protected override async onInitialize(options: AiProviderRegisterOptions): Promise<void> { | ||
| if (options.mode === "inline") { | ||
| const { env } = await import("@sroussey/transformers"); | ||
| // @ts-ignore -- backends.onnx.wasm.proxy is not fully typed | ||
| env.backends.onnx.wasm.proxy = true; | ||
| } | ||
| } | ||
|
|
||
| override async dispose(): Promise<void> { | ||
| if (this.tasks) { | ||
| const { clearPipelineCache } = await import("./common/HFT_JobRunFns"); | ||
| clearPipelineCache(); | ||
| } | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This getting-started snippet registers HuggingFaceTransformersProvider in inline mode without passing the task map (
HFT_TASKS). With the new provider API, inline mode requires tasks injected via the constructor. Update the snippet to passHFT_TASKS(or change the doc to demonstrate worker-mode registration).