Standardize certain (Dagster Cloud) environment variables as a public API

[danielgafni]

What's the use case?

There are a bunch of important environment variable typically set in Dagster Cloud that are often useful for users, such as:

• DAGSTER_CLOUD_DEPLOYMENT_NAME
• DAGSTER_CLOUD_GIT_SHA

and others.

DagsterRun already exposes them as dagster_execution_info property (btw, should this be added to the public API docs?).

DagsterRun.dagster_execution_info is often used by remote execution components such as Pipes, Executors, Run Launchers, etc (not all of them are using this attribute yet, but they really should).

For example, DagsterRun.dagster_execution_info["dagster/git-sha"] (which is taken from DAGSTER_CLOUD_GIT_SHA) is useful for me in dagster-ray because I can easily identify RayCluster objects that have the right version of my source code already deployed (for cluster sharing). DAGSTER_CLOUD_DEPLOYMENT_NAME is useful as well.

But right now there is no standard way to provide this and other values to DagsterRun.dagster_execution_info outside of Dagster Cloud: if I'd want to self-host Dagster, I would still have to set the same DAGSTER_CLOUD_ environment variables to do it.

This might be just a naming problem, but it would be great if Dagster had a standard, reliable public API for these environment variables that is independent from Dagster Cloud. Simply removing the _CLOUD should be good enough.

Ideas of implementation

Start using DAGSTER_DEPLOYMENT_NAME, DAGSTER_GIT_SHA and other environment variables (along with the legacy _CLOUD_ variants for backcompat). Document this in the API docs.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[danielgafni]

Curious what @gibsondan has to say about this :)