Skip to content

feat(datasource): unify default embedding model on EmbeddingGemma #1422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vkehfdl1 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(datasource): unify default embedding model on EmbeddingGemma #1422

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ Or through the trusted config factory:
"datasources": {
"discord": {
"instanceId": "community",
"connector": { "source": "wiretap", "embeddingModel": "bge-m3", "defaultMode": "hybrid" }
"connector": { "source": "wiretap", "embeddingModel": "embeddinggemma", "defaultMode": "hybrid" }
}
}
}
Expand All @@ -247,7 +247,7 @@ Or through the trusted config factory:
Two defaults are deliberate and worth keeping:

- **`defaultMode: "hybrid"`** — discrawl's FTS index strips newlines without substituting a space, welding words across line breaks into a single unsearchable token (measured at ~47% of post-newline words on a real archive). Semantic recall covers that gap. See [#1413](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1413).
- **`embeddingModel: "bge-m3"`** — semantic search requires an embedding provider (`ollama serve && ollama pull bge-m3`). Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).
- **`embeddingModel: "embeddinggemma"`** — semantic search requires an embedding provider (`ollama serve && ollama pull embeddinggemma`). EmbeddingGemma (Gemma 3 300M, 768-dim, 100+ languages) is the same model family katok uses for KakaoTalk, so all CLI-backed datasources share one local embedder. Do **not** use `nomic-embed-text`: it is English-only and collapses non-English text into one narrow similarity band, silently degrading semantic search to noise. AutoRAG emits a diagnostic when an English-only model is configured. See [#1414](https://github.com/Marker-Inc-Korea/AutoRAG/issues/1414).

A datasource skill should provide polling/cron metadata for routine indexing, source descriptions for the agent prompt, slash-hierarchical opaque source paths such as `/kakao/personal/chunks/<chunk-id>`, and permission tags that match your server-side access policy.

Expand Down
2 changes: 1 addition & 1 deletion scripts/manual-qa/run-qa-discrawl-live.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
* Setup (wiretap — no token, reads the local Discord Desktop cache):
* 1. brew install openclaw/tap/discrawl
* 2. Have the Discord desktop app installed and signed in at least once.
* 3. (semantic) brew install ollama && ollama serve && ollama pull bge-m3
* 3. (semantic) brew install ollama && ollama serve && ollama pull embeddinggemma
*
* Setup (bot API — ToS-sanctioned automation):
* 1. https://discord.com/developers/applications -> New Application -> Bot
Expand Down
9 changes: 5 additions & 4 deletions src/datasource/skills/discrawl/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,11 +50,12 @@ export const ENGLISH_ONLY_EMBEDDING_MODELS: ReadonlySet<string> = new Set([
]);

/**
* Default embedding model AutoRAG configures for discrawl. `bge-m3` covers
* 100+ languages at 1024 dimensions and supports cross-lingual retrieval, so
* mixed-language Discord servers work without per-deployment tuning.
* Default embedding model AutoRAG configures for discrawl. `embeddinggemma`
* (Gemma 3 300M, 768 dimensions) covers 100+ languages and matches the
* EmbeddingGemma embedder katok uses for KakaoTalk, so every CLI-backed
* datasource shares one local embedding model (served through Ollama).
*/
export const DEFAULT_DISCRAWL_EMBEDDING_MODEL = "bge-m3";
export const DEFAULT_DISCRAWL_EMBEDDING_MODEL = "embeddinggemma";

/**
* Configuration for the discrawl client. All fields optional; defaults mirror
Expand Down
5 changes: 4 additions & 1 deletion test/datasource/skills/discrawl/skill.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import type {
DiscrawlSearchResult,
DiscrawlSyncResult,
} from "../../../../src/datasource/skills/discrawl/types.ts";
import { DEFAULT_DISCRAWL_EMBEDDING_MODEL } from "../../../../src/datasource/skills/discrawl/types.ts";

function okDoctor(overrides: Partial<Record<string, boolean | string>> = {}): DiscrawlDoctorResult {
return {
Expand Down Expand Up @@ -164,7 +165,9 @@ describe("DiscrawlSkill index", () => {
embeddingModel: "nomic-embed-text",
}).index();
expect(result.ok).toBe(true);
expect(result.diagnostics.some((d) => d.message.includes("English-only"))).toBe(true);
const warning = result.diagnostics.find((d) => d.message.includes("English-only"));
expect(warning).toBeDefined();
expect(warning?.message).toContain(DEFAULT_DISCRAWL_EMBEDDING_MODEL);
});

it("does not warn for a multilingual embedding model", async () => {
Expand Down
Loading