[feat] Add LlamaCpp provider and integration tests

[sroussey]

• Introduced the LlamaCpp provider for running GGUF models locally using node-llama-cpp.
• Added necessary constants, job run functions, and model schema for LlamaCpp integration.
• Implemented integration tests for LlamaCpp provider, covering model downloading, text generation, and embedding tasks.
• Updated package.json files to include node-llama-cpp as a dependency across relevant packages.
• Enhanced .gitignore to exclude model files and added test setup configuration in vitest.

[sroussey]

closes #208

[Copilot]

Pull request overview

Adds a new LOCAL_LLAMACPP AI provider to run GGUF models locally via node-llama-cpp, and introduces an integration test suite that exercises real model download + generation + embeddings.

Changes:

• Added provider-llamacpp implementation (constants, model schema, task run fns, provider class, worker registration, exports).
• Added node-llama-cpp dependency wiring and package export entrypoints.
• Added integration tests that download small GGUF models from HuggingFace and run end-to-end workflows.

Reviewed changes

Copilot reviewed 11 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/test/src/test/ai-provider/LlamaCppProviderIntegration.test.ts	New real-model integration tests for LlamaCpp download/generation/embedding.
packages/test/package.json	Adds node-llama-cpp for the test package.
packages/ai-provider/src/provider-llamacpp/index.ts	Public barrel exports for the LlamaCpp provider package entrypoint.
packages/ai-provider/src/provider-llamacpp/common/LlamaCpp_ModelSchema.ts	JSON schema + TS types for LlamaCpp model records/config.
packages/ai-provider/src/provider-llamacpp/common/LlamaCpp_JobRunFns.ts	Core implementation for Download/Unload/Generation/Embedding + streaming support.
packages/ai-provider/src/provider-llamacpp/common/LlamaCpp_Constants.ts	Declares LOCAL_LLAMACPP and default models dir constant.
packages/ai-provider/src/provider-llamacpp/LlamaCpp_Worker.ts	Worker-side registration function for LlamaCpp tasks.
packages/ai-provider/src/provider-llamacpp/LlamaCppProvider.ts	Provider class definition and supported task list.
packages/ai-provider/src/index.ts	Re-exports LlamaCpp constants/schema/provider from the main package entrypoint.
packages/ai-provider/package.json	Adds ./llamacpp export + adds node-llama-cpp as optional peer + dev dependency.
package.json	Adds node-llama-cpp to the workspace catalog.
bun.lock	Locks node-llama-cpp and transitive dependencies.
.gitignore	Ignores models/ directory used for downloaded GGUF test artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This integration suite downloads real models (~120MB total) and runs native inference. As written it will run under bun test in CI (see .github/workflows/test.yml) and is very likely to time out or fail on runners without node-llama-cpp native prerequisites / network access. Consider guarding the whole describe with a runtime condition (e.g., describe.skipIf(process.env.CI || !process.env.RUN_LLAMACPP_INTEGRATION)), so it only runs when explicitly enabled.

Suggested change

	describe("LlamaCpp Integration (real models, no mocks)", () => {
	describe.skipIf(process.env.CI || !process.env.RUN_LLAMACPP_INTEGRATION)("LlamaCpp Integration (real models, no mocks)", () => {

[Copilot]

This test assumes both models were downloaded by previous tests (“Both models already downloaded…”). That makes the file order-dependent and can fail when running only this test (filtered runs) or if the runner changes ordering. Prefer downloading/ensuring the models in beforeAll (or within this test) and treating cache-hit behavior as an optional assertion rather than a prerequisite.

[Copilot]

LlamaCpp_Download receives an AbortSignal but never uses it, so aborting a DownloadModelTask won’t actually stop the underlying download. This is inconsistent with existing provider behavior (see DownloadModelAbort.test.ts) and can lead to runaway downloads. Wire signal into the download process (e.g., race downloader.download() with an abort promise and call the downloader’s cancel/abort API if available), and ensure the interval is cleared on abort.

Suggested change

	try {
	modelPath = await downloader.download();
	} finally {
	clearInterval(progressInterval);
	let abortHandler: (() => void) | undefined;
	try {
	const downloadPromise = downloader.download();
	const racedPromise: Promise<string> =
	signal
	? Promise.race<string>([
	downloadPromise as Promise<string>,
	new Promise<never>((_, reject) => {
	const onAbort = () => {
	// Attempt to cancel/abort the underlying downloader if supported
	try {
	const anyDownloader = downloader as any;
	if (typeof anyDownloader.cancel === "function") {
	anyDownloader.cancel();
	} else if (typeof anyDownloader.abort === "function") {
	anyDownloader.abort();
	}
	} catch {
	// Ignore errors from cancellation attempts
	}
	const err = new Error("The download was aborted.");
	(err as any).name = "AbortError";
	reject(err);
	};
	abortHandler = onAbort;
	if (signal.aborted) {
	onAbort();
	return;
	}
	signal.addEventListener("abort", onAbort);
	}),
	])
	: (downloadPromise as Promise<string>);
	modelPath = await racedPromise;
	} finally {
	clearInterval(progressInterval);
	if (signal && abortHandler) {
	signal.removeEventListener("abort", abortHandler);
	}

[Copilot]

This provider currently doesn’t override dispose(), but the implementation uses module-level caches (models, textContexts, embeddingContexts, llamaInstance). Without a provider-level dispose that calls disposeLlamaCppResources(), inline-mode usage can leak memory across runs. Suggest overriding dispose() (similar to HuggingFaceTransformersProvider) to dynamically import and invoke disposeLlamaCppResources() when tasks are present.

Suggested change

	constructor(
	tasks?: Record<string, AiProviderRunFn<any, any, LlamaCppModelConfig>>,
	streamTasks?: Record<string, AiProviderStreamFn<any, any, LlamaCppModelConfig>>
	) {
	super(tasks, streamTasks);
	}
	/** Tracks whether this provider was constructed with inline-mode tasks. */
	private readonly hasInlineTasks: boolean;
	constructor(
	tasks?: Record<string, AiProviderRunFn<any, any, LlamaCppModelConfig>>,
	streamTasks?: Record<string, AiProviderStreamFn<any, any, LlamaCppModelConfig>>
	) {
	super(tasks, streamTasks);
	this.hasInlineTasks = !!(tasks || streamTasks);
	}
	override async dispose(): Promise<void> {
	if (this.hasInlineTasks) {
	try {
	const { disposeLlamaCppResources } = await import("./common/LlamaCpp_Runtime");
	if (typeof disposeLlamaCppResources === "function") {
	await disposeLlamaCppResources();
	}
	} catch {
	// Ignore cleanup errors to avoid breaking provider teardown.
	}
	}
	await super.dispose();
	}