checkpointed confluence #4473
Merged
checkpointed confluence #4473
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR integrates a new checkpointing mechanism for multiple connectors, replacing CheckpointConnector with the new CheckpointedConnector interface and standardizing checkpoint management across the codebase.
- Updated backend/onyx/connectors/factory.py to support CheckpointedConnector for POLL input types.
- Modified backend/onyx/connectors/connector_runner.py to enforce time_range and wrap outputs with CheckpointOutputWrapper.
- Refactored backend/onyx/background/indexing/checkpointing_utils.py for updated checkpoint validation and loading.
- Adjusted key connectors (Confluence, GitHub, GoogleDrive, etc.) to increment checkpoints per batch with enhanced error propagation.
- Test files now use utilities like load_all_docs_from_checkpoint_connector for thorough validation.
15 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile
evan-danswer
force-pushed
the
confluence-checkpointed-connector
branch
from
9156b74 to
6df1008
Compare
evan-danswer
force-pushed
the
confluence-checkpointed-connector
branch
from
6df1008 to
11854b1
Compare
Weves
approved these changes
Apr 14, 2025
| if not self.continue_on_failure: | ||
| if _should_propagate_error(e): | ||
| raise | ||
| # TODO: should we remove continue_on_failure entirely now that we have checkpointing? |
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
evan-danswer
force-pushed
the
confluence-checkpointed-connector
branch
from
11a806c to
5b5c35b
Compare
alex-usher
pushed a commit
to dialin-ai/dialin-search
that referenced
this pull request
Apr 17, 2025
* checkpointed confluence * confluence checkpointing tested * fixed integration tests * attempt to fix connector test flakiness * fix rebase
alex-usher
added a commit
to dialin-ai/dialin-search
that referenced
this pull request
Apr 17, 2025
* rollback properly on exception (onyx-dot-app#4073) * rollback properly on exception * rollback on exception * don't continue if we can't set the search path * cleaner handling via context manager --------- Co-authored-by: Richard Kuo (Danswer) <[email protected]> * cohere validation logic update (onyx-dot-app#4523) * fix font for dark mode (onyx-dot-app#4527) * Fix onyx-dot-app#3764: Dynamically handle default branch in GitLab connector * Fix black * k (onyx-dot-app#4525) * checkpointed confluence (onyx-dot-app#4473) * checkpointed confluence * confluence checkpointing tested * fixed integration tests * attempt to fix connector test flakiness * fix rebase * extra logging for uncommon permissions cases (onyx-dot-app#4532) * extra logging for uncommon permissions cases * address CW comments * Feature/celery beat watchdog (onyx-dot-app#4534) * upgrade celery to release version * make the watchdog script more reusable * use constant * code review * catch interrupt --------- Co-authored-by: Richard Kuo (Onyx) <[email protected]> * Miscellaneous cleanup (onyx-dot-app#4516) * stricter typing * k * Small no-letsencrypt improvement * Remove alert upon submission (onyx-dot-app#4537) * Fix confluence perm sync ancestry (onyx-dot-app#4536) * Fix confluence perm sync ancestry * Address EL comments * add test for special case * remove print * Fix test * Fix bug with saml validation (onyx-dot-app#4522) * fix bug with saml validation * k * Changes base containers to dialin AI * add 4.1 (onyx-dot-app#4540) * fix re-processing of previously seen docs Confluence (onyx-dot-app#4544) * fix re-processing of previously seen docs * performance * Skip image files for S3 (onyx-dot-app#4535) * skip image files * process images s3 * tests * k * update * nit * update * rollback properly on exception (onyx-dot-app#4073) * rollback properly on exception * rollback on exception * don't continue if we can't set the search path * cleaner handling via context manager --------- Co-authored-by: Richard Kuo (Danswer) <[email protected]> * cohere validation logic update (onyx-dot-app#4523) * Fix onyx-dot-app#3764: Dynamically handle default branch in GitLab connector * Fix black * k (onyx-dot-app#4525) * fix font for dark mode (onyx-dot-app#4527) * checkpointed confluence (onyx-dot-app#4473) * checkpointed confluence * confluence checkpointing tested * fixed integration tests * attempt to fix connector test flakiness * fix rebase * extra logging for uncommon permissions cases (onyx-dot-app#4532) * extra logging for uncommon permissions cases * address CW comments * Feature/celery beat watchdog (onyx-dot-app#4534) * upgrade celery to release version * make the watchdog script more reusable * use constant * code review * catch interrupt --------- Co-authored-by: Richard Kuo (Onyx) <[email protected]> * Miscellaneous cleanup (onyx-dot-app#4516) * stricter typing * k * Small no-letsencrypt improvement * Remove alert upon submission (onyx-dot-app#4537) * Fix confluence perm sync ancestry (onyx-dot-app#4536) * Fix confluence perm sync ancestry * Address EL comments * add test for special case * remove print * Fix test * Fix bug with saml validation (onyx-dot-app#4522) * fix bug with saml validation * k * add 4.1 (onyx-dot-app#4540) * fix re-processing of previously seen docs Confluence (onyx-dot-app#4544) * fix re-processing of previously seen docs * performance * Skip image files for S3 (onyx-dot-app#4535) * skip image files * process images s3 * tests * k * update * nit * update * Changes base containers to dialin AI --------- Co-authored-by: rkuo-danswer <[email protected]> Co-authored-by: Richard Kuo (Danswer) <[email protected]> Co-authored-by: pablonyx <[email protected]> Co-authored-by: evan-danswer <[email protected]> Co-authored-by: ThomaciousD <[email protected]> Co-authored-by: Weves <[email protected]> Co-authored-by: Raunak Bhagat <[email protected]> Co-authored-by: Chris Weaver <[email protected]> Co-authored-by: Alex Usher <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Addresses https://linear.app/danswer/issue/DAN-1703/checkpointed-confluence-connector
Checkpointing for the confluence connector. Creates a new checkpoint each time the previous version of the connector would have finished a batch.
How Has This Been Tested?
tested in UI and unit tests
Backporting (check the box to trigger backport action)
Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.