Skip to content

[DRAFT] Chunker #2260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[DRAFT] Chunker #2260

wants to merge 1 commit into from

Conversation

atroyn
Copy link
Contributor

@atroyn atroyn commented May 30, 2024 •
edited
Loading

Description of changes

This PR adds the type, and a default implementation for built-in chunkers for Chroma. The default text chunker is a riff on LangChain's RecursiveTextSplitter but refactored a bit for our purposes.

The API surface is designed to resemble the EmbeddingFunction implementation, with the exception that they are not a property of Collections.

Test plan

TBC

Documentation Changes

TBC

TODO

  • Tests
  • Docs

@vercel Vercel
Copy link

vercel bot commented May 30, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
chroma ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jun 6, 2024 4:20am

@github-actions GitHub Actions
Copy link

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

@github-actions GitHub Actions
Copy link

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

@atroyn Graphite App
Copy link
Contributor Author

atroyn commented May 30, 2024

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @atroyn and the rest of your teammates on Graphite Graphite

@atroyn atroyn mentioned this pull request May 30, 2024
Draft
2 tasks
@atroyn atroyn changed the title anton/chunker [Draft] Chunker May 30, 2024
@atroyn atroyn changed the title [Draft] Chunker [DRAFT] Chunker May 30, 2024
@atroyn atroyn marked this pull request as ready for review May 30, 2024 04:09
@atroyn atroyn marked this pull request as draft May 30, 2024 04:26
@atroyn atroyn force-pushed the 05-29-anton_chunker branch from d2b2d7f to 719c37c Compare May 31, 2024 19:56
@github-actions GitHub Actions
Copy link

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

@atroyn atroyn force-pushed the 05-29-anton_chunker branch from 719c37c to 773385a Compare June 3, 2024 20:45
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 3, 2024

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

@atroyn atroyn force-pushed the 05-29-anton_chunker branch from 773385a to d673865 Compare June 4, 2024 18:34
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 4, 2024

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

@atroyn atroyn linked an issue Jun 4, 2024 that may be closed by this pull request
[New Feature][Ease of Use] Smart Chunker #2281
Open
@atroyn atroyn force-pushed the 05-29-anton_chunker branch from d673865 to 205c06c Compare June 5, 2024 18:49
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 5, 2024

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

@atroyn atroyn force-pushed the 05-29-anton_chunker branch from 205c06c to 514cf4d Compare June 6, 2024 04:17
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 6, 2024

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[New Feature][Ease of Use] Smart Chunker
1 participant
@atroyn