[BUG]: Watched GitLab documents from an http self-hosted instance never re-sync

[marmar9615-cloud]

How are you running AnythingLLM?

All versions

What happened?

A GitLab repo imported from a self-hosted instance served over plain http imports fine, but a file from it that is set to watch never re-syncs. On every sync run the collector requests the repo over https, the request fails, and after five failed runs in a row the server removes the document from the watched set.

I expected the re-sync to use the scheme the repo was imported with, which is what the GitHub and Gitea connectors do.

Code path, line numbers at eb7df1e:

1. collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js:62 accepts http or https for a self-hosted URL, and line 72 sets apiBase from the URL origin. The import runs over http and succeeds.
2. collector/utils/extensions/RepoLoader/GitlabRepo/index.js:160-170 generateChunkSource writes projectId, branch, path and pat into the encrypted payload. It does not write the scheme.
3. collector/extensions/resync/index.js:139 resyncGitlab rebuilds the repo URL as `https:${source.pathname}`. The chunkSource is gitlab://http://gitlab.internal:8080/group/project?payload=.... new URL() reads http as the host and leaves //gitlab.internal:8080/group/project as the pathname, so the rebuilt URL is https://gitlab.internal:8080/group/project.
4. fetchGitlabFile builds a loader from that URL. apiBase becomes https://gitlab.internal:8080, and /api/v4/user, .../repository/branches and .../repository/files/<path>/raw all go to https. Against a plain-http server each fetch fails with ERR_SSL_WRONG_VERSION_NUMBER, the handler logs Failed to sync GitLab file content. Target file returned a null content response. and answers { success: false, content: null }.
5. server/jobs/sync-watched-documents.js:79-83 counts the failed runs and calls DocumentSyncQueue.unwatch once they reach maxRepeatFailures, which is 5 (server/models/documentSyncQueue.js:39).

The other two repo connectors already keep the scheme. GithubRepo/index.js:154 stores scheme in the payload and resync/index.js:105 restores it with source.searchParams.get("scheme") || "https" (#6117). GiteaRepo/index.js:154 stores it and resync/index.js:174 restores it (#6087). GitLab is the only one that still hard-codes https.

Affected: v1.16.0, v1.16.1 and master. GitLab documents became watchable in #2817 (resolves #2810), but that change was server side only and the collector had no gitlab key in RESYNC_METHODS, so from then until v1.16.0 every GitLab re-sync failed at dispatch with Type "gitlab" is not a valid type to sync. (collector/extensions/index.js:22). #6087 added resyncGitlab in commit cd139bff ("add gitlab tracking as ad-hoc fix", squashed into 8f97d90f), copying the https:${source.pathname} line out of resyncGithub. #6117 then fixed that line for GitHub only. So GitLab re-sync has worked for https instances and failed for http ones since v1.16.0.

I have a fix that follows the #6117 convention (store scheme in the payload, restore it on resync with a https default for older chunkSources) and will open a PR.

Are there known steps to reproduce?

1. Have a GitLab instance reachable over plain http, for example http://gitlab.internal:8080.
2. Workspace settings > Data Connectors > GitLab Repo. Enter http://gitlab.internal:8080/group/project as the repo URL, a PAT if the project is private, and submit. The import succeeds and the files show up in the document picker.
3. Add one of the files to a workspace and turn on watch for it.
4. Wait for the watched-documents worker (DOCUMENT_SYNC_STALE_AFTER_MS lowers the wait, minimum one hour). Every run fails the same way. The collector output below is not from a deployment: it is from a local run of the real resync.gitlab handler against a stub GitLab API on http, the scripted run described further down. Trimmed, and the first line only appears when a PAT is set:

Invalid Gitlab Access Token provided! Access token will not be used fetch failed
RepoLoader.fetchNextPage [TypeError: fetch failed] { [cause]: ... code: 'ERR_SSL_WRONG_VERSION_NUMBER' }
[Gitlab Loader]: Branch not set! Auto-assigning to a default branch.
RepoLoader.fetchSingleFileContents [TypeError: fetch failed] { [cause]: ... code: 'ERR_SSL_WRONG_VERSION_NUMBER' }
Error: Failed to sync GitLab file content. Target file returned a null content response.

The server then logs, once per run (server/jobs/sync-watched-documents.js:88):

Failed to get a new content response from collector for source ... Skipping, but will retry next worker interval. Attempt 1/5

5. After the fifth failed run the server logs has failed to refresh 5 times continuously and will now be removed from the watched document set. (line 81) and the document is no longer watched.

The same run, scripted, so it does not need an hour or a GitLab install: a stub GitLab v4 API listening on http only, the real loadGitlabRepo, a real EncryptionWorker, the real resync.gitlab handler, and every outbound fetch URL logged. The collector lines in step 4 come from this run.

=== loadGitlabRepo(http://127.0.0.1:61875/group/project)
{"success":true,"reason":null,"data":{"author":"group","repo":"project","projectId":"group%2Fproject","branch":"main","files":1,"destination":"group-project-main-85f6"}}
chunkSource prefix: gitlab://http://127.0.0.1:61875/group/project
payload params: {"projectId":"group/project","branch":"main","path":"README.md","pat":"glpat-x"}
URL parse of chunkSource: host=http pathname=//127.0.0.1:61875/group/project

=== resync.gitlab({chunkSource})
result: {"success":false,"content":null}
outbound URLs attempted during resync:
  https://127.0.0.1:61875/api/v4/user
  https://127.0.0.1:61875/api/v4/projects/group%2Fproject/repository/branches?per_page=100&page=1
  https://127.0.0.1:61875/api/v4/projects/group%2Fproject/repository/files/README.md/raw?ref=master
requests that reached the http server during resync: 0

The ref=master is a second symptom of the same thing: the branch list could not be fetched over https either, so the loader fell back to a branch the repo does not have.

LLM Provider & Model (if applicable)

Not applicable.

Embedder Provider & Model (if applicable)

Not applicable.