Skip to content

fix: keep the GitLab repo scheme so http self-hosted instances can re-sync - #6294

Open
marmar9615-cloud wants to merge 1 commit into
Mintplex-Labs:masterfrom
marmar9615-cloud:fix/gitlab-resync-scheme
Open

fix: keep the GitLab repo scheme so http self-hosted instances can re-sync#6294
marmar9615-cloud wants to merge 1 commit into
Mintplex-Labs:masterfrom
marmar9615-cloud:fix/gitlab-resync-scheme

Conversation

@marmar9615-cloud

Copy link
Copy Markdown
Contributor

Pull Request Type

  • 🐛 fix (Bug fix)

Relevant Issues

resolves #6293

Description

A watched document imported from a self-hosted GitLab served over plain http never re-syncs. resyncGitlab rebuilt the repo URL as https: plus the stored pathname, and the chunkSource payload had no scheme to restore. Every sync run of an http://gitlab.internal:8080/group/project document requested https://gitlab.internal:8080/..., the fetch failed, and after five failed runs the server dropped the document from the watched set.

The import works over http, since the loader accepts either scheme and builds apiBase from the URL origin. Only re-sync lost it: new URL("gitlab://http://host:8080/group/project?payload=...") reads http as the host and leaves //host:8080/group/project as the pathname, and resyncGitlab put https: in front of that.

The fix is the convention #6117 added for GitHub and #6087 shipped with for Gitea, transposed to GitLab:

  • collector/utils/extensions/RepoLoader/GitlabRepo/index.js: generateChunkSource stores scheme: new URL(repo.repo).protocol.replace(":", "") in the encrypted payload. The function is now exported so the test can build a real chunkSource, as GithubRepo/index.js does.
  • collector/extensions/resync/index.js: resyncGitlab rebuilds the URL as `${source.searchParams.get("scheme") || "https"}:${source.pathname}`.

ChunkSources written before this change have no scheme and keep defaulting to https, so existing https documents are unaffected. A document already imported over http keeps its old chunkSource, still resyncs over https, and has to be re-imported once to pick the scheme up. #6117 has the same limitation for GitHub.

Additional Information

collector/__tests__/utils/extensions/RepoLoader/GitlabRepo.test.js is new: 24 tests in seven describe blocks, the same layout as GithubRepo.test.js and GiteaRepo.test.js. Five run a chunkSource from generateChunkSource through a real EncryptionWorker and the real resyncGitlab. Four of those assert that the URL the user entered is the URL fetchGitlabFile receives on re-sync (https with a PAT, http on a port, a legacy payload without scheme, public gitlab.com). The fifth asserts the { success: false, content: null } response on a failed fetch.

Which case fails where, same test file each time:

Tree Result
Base eb7df1e8, both files 4 failed, 20 passed. Base does not export generateChunkSource, so the four tests that build one through the loader cannot run; the fifth builds its payload by hand and passes.
Base resync/index.js, chunkSource change kept 1 failed: an http self-hosted host keeps its scheme and port through a resync. fetchGitlabFile received https://gitlab.example.com:8080/acme/widgets.
Wrong fix: parse the scheme out of source.pathname 1 failed, the same http case. The pathname is //gitlab.example.com:8080/acme/widgets; the scheme is not in it.
Wrong fix: drop the || "https" default 1 failed: a chunkSource stored before the protocol was recorded still resyncs over https. The repo URL became null://gitlab.com/acme/widgets.
This PR 24 passed.

yarn test from the repo root, three runs: 55 suites, 772 tests, against 54 suites and 748 tests on this base. server/__tests__/utils/helpers/modelPricing/index.test.js failed one of the three runs and passes on its own; it is untouched by this change.

I did not run yarn lint at the root, which also lints the frontend. eslint and prettier --check are clean on the changed files; the collector eslint config ignores __tests__/**.

Not in this PR: resyncConfluence has the same https:${source.pathname} line (resync/index.js:67) but fails elsewhere. Its requests go through the stored baseUrl, which keeps its scheme, and each doc.metadata.url is built from it (ConfluenceLoader/index.js:134). pageUrl is only compared, doc.metadata.url === pageUrl (Confluence/index.js:217), so on an http instance the pages load and the match fails with "Target page could not be found in Confluence space." That is a comparison fix, not a payload fix, so it belongs in its own change.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

…-sync

resyncGitlab rebuilt the repo URL as "https:" plus the stored pathname, and
the GitLab chunkSource never recorded the scheme, so a watched document from
a self-hosted GitLab served over http was fetched over https on every sync
run and un-watched after five failures. Store the scheme in the chunkSource
payload and restore it on resync, defaulting to https for chunkSources
written before this change, which is the convention the GitHub and Gitea
connectors already use.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Watched GitLab documents from an http self-hosted instance never re-sync

1 participant