feat(git-providers): support docker url and http connections for git providers

[unleashit]

This addresses #1591 to allow Gitea and Gitlab (not yet implemented) providers to use local and/or http connections.

• A new field was added to the db and gitea provider forms called giteaUrlAlt
• Added an accordion drop down called "advanced options" with description for the fields
• For serverside fetch requests to the provider (authorization/refresh/test/repo actions that use fetch), if giteaUrlAlt is set that will be used instead of giteaUrl

So far Gitea should be complete, although I could use some advice for how to build and use the docker container locally so I can test the new feature using a docker url. I tried running pnpm docker:build:canary and got errors, and the Dockerfile at the root seems to want to work with /usr/src/app.

[Siumauricio]

I wonder if we really always apply this replace regardless of one property or the other, I think it would not be necessary to maintain another property, since we apply the same operation for both properties.

[unleashit]

Thanks for reviewing. Not sure I understand the question. That regex simply removes any trailing slashes in case the user added them, which avoids a possible duplicate (the next line is const url = ${baseUrl}/api/v1/user/repos;` which hardcodes a slash. I didn't write that, it was already in the code. Are you suggesting to strip it off when the user first submits the form? That could make sense as long as we know it won't affect anything else.

As far as maintaining the new property (I think you mean the new database field giteaUrlAlt right?), yes I think its the only way to achieve this PR. In my case, I need two separate URLs:

1. http://localhost:port - this is needed to bring up the authorization page in the browser or perform git commands that node spawns like git clone
2. http://gitea:3000 - this is the new field/property that is needed for fetch requests if Dokploy is running from a container and can't reach the above url

Again, this is only needed in the case where Gitea is not exposed to the internet.

If you can think of a good way to achieve this some other way I'd be happy to try it. Otherwise if I'm making sense, I/we still need to test this with Dokploy running from a container. I'm not sure if the current NPM commands are up to date on that.

[unleashit]

Just looked again and noticed that the trailing slash stripping isn't always done. Like:

https://github.com/Dokploy/dokploy/blob/canary/packages/server/src/utils/providers/gitea.ts#L57

Again, these were already there (I think in the diff because I changed the var names).

If you want to make it more consistent, I think either we should strip them before adding to the database or possibly remove it and leave it to the user to discover the error

[Siumauricio]

Yes but I mean in this operation you basically say if this property doesn't exist use the other one and apply the same replace operation, so really you will always apply the same operation for both properties, I don't see any other place where you really do a different operation than giteaUrl either.

[unleashit]

Sorry, I'm still not understanding if you're only taking issue with the actual replace command that removes a possible trailing slash, or the bigger picture of adding and using the new field giteaUrlAlt.

If the former, again, I just kept what was already there. But it's trivial to change it so it matches how the urls are constructed in other places if you like. But I think it sounds like you're questioning more than just that.

I don't see any other place where you really do a different operation than giteaUrl either.

I'm not 100% sure what you're saying here, because there are differences. If the user defines a giteaUrlAlt in the advanced options, it will get used for certain operations, but not for others in packages/server/src/utils/providers/gitea.ts.

Imagine we have these two urls:

giteaUrl: http://localhost:3000
giteaUrlAlt: http://giteea:3000

For operations where node spawns a shell to run commands like git clone, the normal giteaUrl is always used because the command is being called from the host. giteaUrl would also be used for the initial oauth browser authorization (you can't launch a browser tab with a docker url).

But likewise, the localhost url won't work in a fetch request (assuming Dokploy is running inside a container, which it normally is in production). Thus the need for two different urls in this situation.

All of this said, if you don't want to add this or want to change it I'd understand. In the cloud version it's going to be redundant.

[unleashit]

Hey, when you have a chance any update on this? I'm not sure why you're saying it's a problem to remove the trailing slash in both cases of user input (giteaUrl or giteaUrlAlt). That's as intended to to avoid duplicate slashes (the next line in the code hard codes a slash). But I can either close this if you don't want it, or we can also limit the scope to at least allow http in the giteaUrl... which would in my case at least allow me to keep using Gitea with Dokploy (although that's a hacky solution that requires manually changing the url after granting auth).

[Siumauricio]

Yes but I mean in this operation you basically say if this property doesn't exist use the other one and apply the same replace operation, so really you will always apply the same operation for both properties, I don't see any other place where you really do a different operation than giteaUrl either.

[unleashit]

Closing as we seemed to have had some communication issues, and I wasn't able to use Dokploy without this fix.

[vitonsky]

Hi @Siumauricio @unleashit, what the cause of this problem? And how you tried to solve it in this PR?

[unleashit]

Only problem is the providers aren't designed to work with local Gitea/Gitlab instances that aren't exposed to the Internet. If you're not in that particular situation, you don't need this.

[vitonsky]

Only problem is the providers aren't designed to work with local Gitea/Gitlab instances that aren't exposed to the Internet. If you're not in that particular situation, you don't need this.

I mean what a technical details?

I've read pull request description and your comments here and i don't understand what root cause you found and trying to solve with changes you've made.

I'm not sure why changes in SQL may be needed here. Could you elaborate what exactly works wrong currently and how you trying to fix it with your changes in code.

I've wrote a blog post about ideal format of pull requests descriptions:

The main rule is comment must be short (1-2 paragraph) and must not repeat task description or steps, instead comment must contain information about root cause of this bug and how this code fix this problem.

[unleashit]

Did you read through the comments here and the original issue? Perhaps the top level description could have had a little more of the "why", but the information is all there.

There are two problems: 1. when the provider makes API calls to Gitea, it hard codes https (if you wanted, you could probably get around this with a self signed cert) and 2 a second user provided URL can be needed (thus an extra DB field) because the host's "localhost" is not exposed to Dokploy. if you simply provide a gitea url like http://localhost:5000 (which IS needed for some things like launching a browser tab with the Gitea route for authorization and also some host spawned shell commands), API calls made from the server to Gitea will fail. This aimed to fix that by adding a new field for a docker version of the url like http://gitea:3000. Assuming Gitea is running in a container AND it was part of the dokploy network, the api calls could then work.

But as you can see, this is all pretty specific, so I'm not exactly surprised it wasn't a priority for the author.

[vitonsky]

@unleashit thanks for explanation. As i understood the problem is currently dokploy uses the same URL for requests on boths sides, server and client browser and in case with setup where Gitea is inaccessible for the internet (for example over VPN) it does not work because different endpoints must be used.

So "root cause" by your opinion is that currently Dokploy supports only single origin for both sides. Is that correct?

My current setup is a public domain that point on private IP.
This private IP is a static IP in Dokploy network like 10.10.10.10.
Gitea is deployed on this network and it is accessible for Dokploy container.
So both, me (with enabled VPN) and Dokploy may resolve and access gitea with IP like 10.10.10.10.

But when I click "Connect to Gitea" i redirect successfully back to Dokploy, but I see a toast error in UI "Gitea Connection Failed - Invalid authorization code or state parameter" Dokploy have no error in logs.

As you can see, even if both sides (client and server) can resolve host properly and can access it, we still have problem. So it looks like found root cause is wrong.

Did you run any tests against your fix? If yes, tell about your steps and results.

[unleashit]

From the sound of that, your situation may or may not be a bit different since Gitea isn't running locally. I'm assuming you've tried this, but if not try attaching a terminal to the Dokploy container to see if it has access. Ex: curl http://10.10.10.10:port. Assuming that "doesn't" work (if it does, you should be good as long as your browser can also tunnel in), there are ways of configuring docker (or just the container/network) to expose a node and/or the host's networking. I'm not on expert on these, and I'm aware they come with security implications.

Other than the forced https fix, this PR pretty much addresses the specific problem of localhost not being exposed to docker. If you give it an IP for Gitea that isn't accessible from the Dokploy container, this will do you no good.

EDIT: on second thought, I don't have much experience with a multi-node docker setup. Maybe this is the type fix for you if docker can route something like http://gitea:5000 to the right IP.

But when I click "Connect to Gitea" i redirect successfully back to Dokploy, but I see a toast error in UI "Gitea Connection Failed - Invalid authorization code or state parameter" Dokploy have no error in logs.

The first part is working because "connect to gitea" launches a browser tab. That loads because the browser is calling it, not Dokploy. The rest fails, including the oAuth handshake because.... Dokploy can't connect.

Didn't write any tests since I didn't notice many tests in this project, but I had it working fine for my use case in production. I've given up on this (Dokploy is an awesome GUI, but doesn't automate as well as I'd like) but you're welcome to take over on it if you like.