Add "from git repository" as a valid code location

[mkleinbort-ic]

What's the use case?

As far as I know, dagster only supports python modules and files as "code locations"

I think there are a lot of workflows that would be enabled by supporting git repositories as valid code locations.

In particular, a team could just merge code into main and the "reload code location" action in Dagster (I envision this as a git pull) would deploy these changes.

Ideas of implementation

In the spirit of https://docs.dagster.io/concepts/code-locations

locations:
  - location_name: my-code-location-from-file
    code_source:
      python_file: my_file.py
      
  - location_name: my-code-location-from-module
    code_source:
      module_name: my_module_name

  - location_name: my-code-location-from-repo
    code_source:
      git_remote: https://github.com/dagster-io/project-dagster-university.git

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[mkleinbort-ic]

On reflection, is this the main use case of running a the dagster-api-grpc server?

https://docs.dagster.io/_apidocs/cli#dagster-api-grpc

It looks to support remote repositories - but this could probably be streamlined as an option in the workspace.yaml

[rexledesma]

Closing because we already support this in Dagster Cloud, through our CI/CD setup with GitHub actions: https://docs.dagster.io/dagster-cloud/getting-started#step-4-configure-cicd-for-your-project.

[mkleinbort-ic]

@rexledesma
Hi, that's not quite what this is asking for, so I wouldn't say it's completed.

The ask is to be able to support any arbitrary git remote provided the server can execute a git-pull - such as an on-prem git server (without the github dependency, or the internet for that matter).

Also not using dagster cloud.

[mkleinbort-ic]

If it helps to clarify the ask:

[mkleinbort-ic]

I found that adding this

def update_from_git_remote():
    try:
        import git, pendulum
        this_repository = git.cmd.Git('.')
        this_repository.pull()
        print(f'Repository refreshed successfully at {pendulum.now()}')
    except Exception as e:
        print(f"Could not update git repository due to {e}")
        
update_from_git_remote()

At the very start of the dagster code location (ie. before the Definition object/loading assets, etc...) works fine.

The workflow is then that whenever the code location is reloaded (either through the UI or on a run-try etc...) the latest version of the codebase is pulled from remote.

But I'm sure there is a better way to support this.

[wiktor-jurek]

I agree that this isn't closed - I appreciate that adding remote code locations may be a USP of dagster+, but it would be good to come to how we can use remote repositories as code locations using the OSS version.