Skip to content

refactor: simply the code #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hustcc merged 4 commits into from
Jul 8, 2026
Merged

refactor: simply the code #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
629f436
refactor: simply the code
hustcc Jul 7, 2026
bd11847
refactor: update reranker
hustcc Jul 7, 2026
6655907
fix: isCJK
hustcc Jul 7, 2026
fa2b2e7
refactor: storage
hustcc Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 25 additions & 35 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ A local context retrieval library that enables semantic search over your documen

## Features

- 📄 **Multi-format Support**: Markdown, JSON, Text 文档自动加载与向量化
- 🔍 **Hybrid Retrieval**: 向量语义 + FTS 全文检索双路召回，RRF 融合排序
- 🔁 **Two-stage Reranking**: KeywordReranker 精排，关键词命中优先
- 🌐 **Query Expansion**: 用户自定义同义词表，CN↔EN 跨语言召回增强
- **Multi-format Loading**: Automatic parsing and vectorization of Markdown, JSON, and plain text files
- **Hybrid Search**: Combines semantic vectors with full-text search using RRF fusion for better recall
- **Two-stage Ranking**: Coarse vector search followed by keyword-based reranking for precision
- **Query Expansion**: Extends queries with user-defined synonym maps for cross-language and domain-specific matching


## Quick Start
Expand All @@ -23,8 +23,8 @@ npm install @antv/context
```typescript
import { Context } from '@antv/context';

// Standard creation — specify vectorsDir
const ctx = await Context.create({ vectorsDir: './vectors' });
// Create context (vectorsDir is optional, defaults to .context/vectors)
const ctx = await Context.create();

// Load documents into a specific library with automatic vectorization
await ctx.load('g2', './g2-docs/**/*.md');
Expand All @@ -45,7 +45,7 @@ await ctx.close();

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `vectorsDir` | `string` | — | **Required**. Directory to store vector files |
| `vectorsDir` | `string` | `.context/vectors` | Directory to store vector files |
| `basePath` | `string` | `process.cwd()` | Base path for resolving document IDs. Set for cross-machine consistent IDs. |
| `onProgress` | `(phase, detail) => void` | — | Progress callback for `load()` phases: `'load'` → `'embed'` → `'insert'`. |
| `queryExpansion` | `QueryExpansionOptions | false` | `false` (no-op) | Query expansion with user-provided synonym map. `false` disables. Without `synonyms`, expansion is a no-op. |
Expand All @@ -57,7 +57,7 @@ await ctx.close();

```typescript
const ctx = await Context.create({
vectorsDir: './vectors',
vectorsDir: '.context/vectors',
// Boost title matches 3x over content matches
ftsFieldWeights: { content: 1, title: 3 },
// More "winner-takes-all" ranking
Expand All @@ -69,7 +69,7 @@ const ctx = await Context.create({

```typescript
const ctx = await Context.create({
vectorsDir: './vectors',
vectorsDir: '.context/vectors',
// Define your own CN↔EN synonym bridges (no built-in defaults)
queryExpansion: {
synonyms: {
Expand All @@ -82,7 +82,7 @@ const ctx = await Context.create({

// Disable query expansion entirely
const ctxNoExpand = await Context.create({
vectorsDir: './vectors',
vectorsDir: '.context/vectors',
queryExpansion: false,
});
```
Expand All @@ -107,7 +107,7 @@ Load phases emit progress via the `onProgress` callback:

```typescript
const ctx = await Context.create({
vectorsDir: './vectors',
vectorsDir: '.context/vectors',
onProgress: (phase, detail) => {
console.log(`${phase}: ${detail.loaded}/${detail.total}`);
},
Expand Down Expand Up @@ -138,9 +138,8 @@ Each result includes:
| `id` | `string` | Document ID |
| `content` | `string` | Document content |
| `score` | `number` | Similarity score (0–1) |
| `scoreMode` | `'vector' | 'hybrid' | 'reranked'` | How the score was computed |
| `meta` | `Record<string, unknown>` | Front-matter metadata (if present) |
| `sourceFilePath` | `string` | Original file path relative to `basePath` |
| `path` | `string` | Original file path relative to `basePath` |


### `ctx.close()`
Expand All @@ -167,13 +166,13 @@ await ctx.close();
+----+-----+ +----+-----+ +----+-----+ +----+-----+
| | | |
+--------------+--------------+ |
v v
+-----------------+ +-----------------+
| FileLoader | | QueryExpander |
| |
+---------v-------+ +-------v----------+
| FileLoader | | QueryExpander |
+--------+--------+ | (SynonymExpander)|
| +--------+--------+
| +--------+---------+
+--------v--------+ |
| EmbedBatch | v
| EmbedBatch | |
+--------+--------+ +--------v--------+
| | Embedder |
+--------v--------+ +--------+--------+
Expand All @@ -182,40 +181,31 @@ await ctx.close();
| Vectorize |
+--------+--------+
|
+--------+-----------+
|
+-----------v-----------+
| |
+-------v-------+ +-------v-------+
+-------v--------+ +-------v-------+
| FTS Text Path | | Vector Path |
|(ftsFieldWeights| | |
+-------+-------+ +-------+-------+
| | | |
+-------+--------+ +-------+-------+
| |
+-----------+-----------+
|
+-----------v-----------+
| RRF Fusion |
| (rankConstant) |
| RRF Fusion |
| (rankConstant) |
+-----------+-----------+
|
+-----------v-----------+
+-----------v------------+
| KeywordReranker |
| (optional, 2nd stage) |
+-----------+-----------+
+-----------+------------+
|
v
Query Result

+------------------------------------------------------------------------+
```

### Module Structure

- **Public API**: `Context`, `QueryOptions`, `QueryResult`, `Document`, `Loader`, `MarkdownLoader`, `JsonLoader`, `TextLoader`, `pathToId`
- **Reranking**: `KeywordReranker`, `createReranker`, `Reranker`, `RerankCandidate`, `RerankResult`, `RerankOptions`
- **Query Expansion**: `SynonymExpander`, `NoopExpander`, `QueryExpander`, `QueryExpansionOptions`
- **Advanced API**: `Embedder`, `TransformersEmbedder`, `EmbedderManager`, `IZvecStore`, `ActualZvecStore`, `Store`


## License

MIT
Loading
Loading