checkpointed confluence #4473
Merged
checkpointed confluence #4473
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR integrates a new checkpointing mechanism for multiple connectors, replacing CheckpointConnector with the new CheckpointedConnector interface and standardizing checkpoint management across the codebase.
- Updated backend/onyx/connectors/factory.py to support CheckpointedConnector for POLL input types.
- Modified backend/onyx/connectors/connector_runner.py to enforce time_range and wrap outputs with CheckpointOutputWrapper.
- Refactored backend/onyx/background/indexing/checkpointing_utils.py for updated checkpoint validation and loading.
- Adjusted key connectors (Confluence, GitHub, GoogleDrive, etc.) to increment checkpoints per batch with enhanced error propagation.
- Test files now use utilities like load_all_docs_from_checkpoint_connector for thorough validation.
15 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile
evan-onyx
force-pushed
the
confluence-checkpointed-connector
branch
from
9156b74 to
6df1008
Compare
evan-onyx
force-pushed
the
confluence-checkpointed-connector
branch
from
6df1008 to
11854b1
Compare
Weves
approved these changes
Apr 14, 2025
| if not self.continue_on_failure: | ||
| if _should_propagate_error(e): | ||
| raise | ||
| # TODO: should we remove continue_on_failure entirely now that we have checkpointing? |
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
evan-onyx
force-pushed
the
confluence-checkpointed-connector
branch
from
11a806c to
5b5c35b
Compare
aronszanto
pushed a commit
to aronszanto/onyx
that referenced
this pull request
Apr 26, 2025
* checkpointed confluence * confluence checkpointing tested * fixed integration tests * attempt to fix connector test flakiness * fix rebase
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Addresses https://linear.app/danswer/issue/DAN-1703/checkpointed-confluence-connector
Checkpointing for the confluence connector. Creates a new checkpoint each time the previous version of the connector would have finished a batch.
How Has This Been Tested?
tested in UI and unit tests
Backporting (check the box to trigger backport action)
Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.