Skip to content

Commit 8f705a2

Browse files
authored
refactor: simply the code (#3)
* refactor: simply the code * refactor: update reranker * fix: isCJK * refactor: storage
1 parent c1d7ea7 commit 8f705a2

33 files changed

Lines changed: 801 additions & 2181 deletions

README.md

Lines changed: 25 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,10 @@ A local context retrieval library that enables semantic search over your documen
88

99
## Features
1010

11-
- 📄 **Multi-format Support**: Markdown, JSON, Text 文档自动加载与向量化
12-
- 🔍 **Hybrid Retrieval**: 向量语义 + FTS 全文检索双路召回,RRF 融合排序
13-
- 🔁 **Two-stage Reranking**: KeywordReranker 精排,关键词命中优先
14-
- 🌐 **Query Expansion**: 用户自定义同义词表,CN↔EN 跨语言召回增强
11+
- **Multi-format Loading**: Automatic parsing and vectorization of Markdown, JSON, and plain text files
12+
- **Hybrid Search**: Combines semantic vectors with full-text search using RRF fusion for better recall
13+
- **Two-stage Ranking**: Coarse vector search followed by keyword-based reranking for precision
14+
- **Query Expansion**: Extends queries with user-defined synonym maps for cross-language and domain-specific matching
1515

1616

1717
## Quick Start
@@ -23,8 +23,8 @@ npm install @antv/context
2323
```typescript
2424
import { Context } from '@antv/context';
2525

26-
// Standard creation — specify vectorsDir
27-
const ctx = await Context.create({ vectorsDir: './vectors' });
26+
// Create context (vectorsDir is optional, defaults to .context/vectors)
27+
const ctx = await Context.create();
2828

2929
// Load documents into a specific library with automatic vectorization
3030
await ctx.load('g2', './g2-docs/**/*.md');
@@ -45,7 +45,7 @@ await ctx.close();
4545

4646
| Parameter | Type | Default | Description |
4747
|-----------|------|---------|-------------|
48-
| `vectorsDir` | `string` | | **Required**. Directory to store vector files |
48+
| `vectorsDir` | `string` | `.context/vectors` | Directory to store vector files |
4949
| `basePath` | `string` | `process.cwd()` | Base path for resolving document IDs. Set for cross-machine consistent IDs. |
5050
| `onProgress` | `(phase, detail) => void` || Progress callback for `load()` phases: `'load'``'embed'``'insert'`. |
5151
| `queryExpansion` | `QueryExpansionOptions | false` | `false` (no-op) | Query expansion with user-provided synonym map. `false` disables. Without `synonyms`, expansion is a no-op. |
@@ -57,7 +57,7 @@ await ctx.close();
5757

5858
```typescript
5959
const ctx = await Context.create({
60-
vectorsDir: './vectors',
60+
vectorsDir: '.context/vectors',
6161
// Boost title matches 3x over content matches
6262
ftsFieldWeights: { content: 1, title: 3 },
6363
// More "winner-takes-all" ranking
@@ -69,7 +69,7 @@ const ctx = await Context.create({
6969

7070
```typescript
7171
const ctx = await Context.create({
72-
vectorsDir: './vectors',
72+
vectorsDir: '.context/vectors',
7373
// Define your own CN↔EN synonym bridges (no built-in defaults)
7474
queryExpansion: {
7575
synonyms: {
@@ -82,7 +82,7 @@ const ctx = await Context.create({
8282

8383
// Disable query expansion entirely
8484
const ctxNoExpand = await Context.create({
85-
vectorsDir: './vectors',
85+
vectorsDir: '.context/vectors',
8686
queryExpansion: false,
8787
});
8888
```
@@ -107,7 +107,7 @@ Load phases emit progress via the `onProgress` callback:
107107

108108
```typescript
109109
const ctx = await Context.create({
110-
vectorsDir: './vectors',
110+
vectorsDir: '.context/vectors',
111111
onProgress: (phase, detail) => {
112112
console.log(`${phase}: ${detail.loaded}/${detail.total}`);
113113
},
@@ -138,9 +138,8 @@ Each result includes:
138138
| `id` | `string` | Document ID |
139139
| `content` | `string` | Document content |
140140
| `score` | `number` | Similarity score (0–1) |
141-
| `scoreMode` | `'vector' | 'hybrid' | 'reranked'` | How the score was computed |
142141
| `meta` | `Record<string, unknown>` | Front-matter metadata (if present) |
143-
| `sourceFilePath` | `string` | Original file path relative to `basePath` |
142+
| `path` | `string` | Original file path relative to `basePath` |
144143

145144

146145
### `ctx.close()`
@@ -167,13 +166,13 @@ await ctx.close();
167166
+----+-----+ +----+-----+ +----+-----+ +----+-----+
168167
| | | |
169168
+--------------+--------------+ |
170-
v v
171-
+-----------------+ +-----------------+
172-
| FileLoader | | QueryExpander |
169+
| |
170+
+---------v-------+ +-------v----------+
171+
| FileLoader | | QueryExpander |
173172
+--------+--------+ | (SynonymExpander)|
174-
| +--------+--------+
173+
| +--------+---------+
175174
+--------v--------+ |
176-
| EmbedBatch | v
175+
| EmbedBatch | |
177176
+--------+--------+ +--------v--------+
178177
| | Embedder |
179178
+--------v--------+ +--------+--------+
@@ -182,40 +181,31 @@ await ctx.close();
182181
| Vectorize |
183182
+--------+--------+
184183
|
185-
+--------+-----------+
186-
|
187184
+-----------v-----------+
188185
| |
189-
+-------v-------+ +-------v-------+
186+
+-------v--------+ +-------v-------+
190187
| FTS Text Path | | Vector Path |
191-
|(ftsFieldWeights| | |
192-
+-------+-------+ +-------+-------+
188+
| | | |
189+
+-------+--------+ +-------+-------+
193190
| |
194191
+-----------+-----------+
195192
|
196193
+-----------v-----------+
197-
| RRF Fusion |
198-
| (rankConstant) |
194+
| RRF Fusion |
195+
| (rankConstant) |
199196
+-----------+-----------+
200197
|
201-
+-----------v-----------+
198+
+-----------v------------+
202199
| KeywordReranker |
203200
| (optional, 2nd stage) |
204-
+-----------+-----------+
201+
+-----------+------------+
205202
|
203+
v
206204
Query Result
207205
208206
+------------------------------------------------------------------------+
209207
```
210208

211-
### Module Structure
212-
213-
- **Public API**: `Context`, `QueryOptions`, `QueryResult`, `Document`, `Loader`, `MarkdownLoader`, `JsonLoader`, `TextLoader`, `pathToId`
214-
- **Reranking**: `KeywordReranker`, `createReranker`, `Reranker`, `RerankCandidate`, `RerankResult`, `RerankOptions`
215-
- **Query Expansion**: `SynonymExpander`, `NoopExpander`, `QueryExpander`, `QueryExpansionOptions`
216-
- **Advanced API**: `Embedder`, `TransformersEmbedder`, `EmbedderManager`, `IZvecStore`, `ActualZvecStore`, `Store`
217-
218-
219209
## License
220210

221211
MIT

0 commit comments

Comments
 (0)