[BUG] fix incorrect read of DATABRICKS_HOST

[ericvergnaud]

Tentatively fix issue

[github-actions]

✅ 41/41 passed, 5 skipped, 22m48s total

_{Running from acceptance #205}

[MrTeale]

@nfx or @sundarshankar89 - This is a blocker for me and my team. Can we get this reviewed and merged?

[asnare]

This is a blocker for me and my team. Can we get this reviewed and merged?

@MrTeale: This was just brought to my attention, and I'm curious about the circumstances under which this is triggered?

[mwojtyczka]

@nfx or @sundarshankar89 - This is a blocker for me and my team. Can we get this reviewed and merged?

@MrTeale Can you please provide info what is the actual scenario you are trying to achieve in which you don't need to provide the host

[mwojtyczka]

LGTM but not sure what is the scenario where host would not be required.

Dismissing till we get clarity on the original issue

[MrTeale]

Apologies all! Appears it was a matter of the DATABRICKS_HOST environment variable not being set correctly. Used an os.environ["DATABRICKS_HOST"] = ... in a notebook and it appears to be working fine.

I do find it odd that when working in a Databricks notebook on a Databricks Personal Compute cluster that these variables aren't automatically created or gathered. Is this normal or should it be gathering these environment variables from another source or means in that context?

[mwojtyczka]

Apologies all! Appears it was a matter of the DATABRICKS_HOST environment variable not being set correctly. Used an os.environ["DATABRICKS_HOST"] = ... in a notebook and it appears to be working fine.

I do find it odd that when working in a Databricks notebook on a Databricks Personal Compute cluster that these variables aren't automatically created or gathered. Is this normal or should it be gathering these environment variables from another source or means in that context?

Do you mind sharing how to do you use the library? pytest and pytester and not meant to be executed in Databricks notebook. Workspace client is auto-authenticated when created within Databricks notebook. So if you want to create WorkspaceClient in Databricks notebook just run ws = WorkspaceClient()

[MrTeale]

Apologies all! Appears it was a matter of the DATABRICKS_HOST environment variable not being set correctly. Used an os.environ["DATABRICKS_HOST"] = ... in a notebook and it appears to be working fine.
I do find it odd that when working in a Databricks notebook on a Databricks Personal Compute cluster that these variables aren't automatically created or gathered. Is this normal or should it be gathering these environment variables from another source or means in that context?

Do you mind sharing how to do you use the library? pytest and pytester and not meant to be executed in Databricks notebook. Workspace client is auto-authenticated when created within Databricks notebook. So if you want to create WorkspaceClient in Databricks notebook just run ws = WorkspaceClient()

Currently we are using a notebook for developers to be able to run these tests within the Databricks UI. Actual test runs will be ran via CI/CD. The main files are as follows including a dummy test.

RunPytest.py:

# Databricks notebook source
dbutils.library.restartPython()

# COMMAND ----------

# MAGIC %pip install pytest pytest-cov databricks-labs-pytester

# COMMAND ----------

import pytest
import sys
import os
from pprint import pprint as print

# COMMAND ----------

os.environ["DATABRICKS_HOST"] = dbutils.secrets.get(scope="PYTEST", key="DATABRICKS_HOST")
os.environ["DATABRICKS_WAREHOUSE_ID"] = dbutils.secrets.get(scope="PYTEST", key="DATABRICKS_WAREHOUSE_ID")
os.environ["DATABRICKS_SERVERLESS_COMPUTE_ID"] = "auto"

# COMMAND ----------

# Skip writing pyc files on a readonly filesystem.
sys.dont_write_bytecode = True

# Run pytest.
retcode = pytest.main(["-rs", "-vvv", "-p", "no:cacheprovider"])

# Fail the cell execution if there are any test failures.
assert retcode == 0, "The pytest invocation failed. See the log for details."

conftest.py:

import logging
from databricks.labs.blueprint.logger import install_logger

install_logger()

logging.getLogger('databricks.labs.pytester').setLevel(logging.DEBUG)

test_dummy:

def test_dummy1(spark, make_catalog, make_schema, make_table):
    from_catalog = make_catalog()
    from_schema = make_schema(catalog_name=from_catalog.name)
    from_table_1 = make_table(catalog_name=from_catalog.name, schema_name=from_schema.name)

This is all based off of this article but to be honest, there's not a lot of good resources out there showcasing how to properly do testing within Databricks' environments. Open to suggestions and guidance in this space

[mwojtyczka]

this article

Thanks for the info. Btw. you should restart python after installing the packages not before.

To improve the experience, we could check the execution context and if pytester is running inside a notebook we could create Workspace Client without any args, as it should be auto-initialized inside a notebook.

For a quick fix to get the host you can run the following in the notebook so that you don't have to store it anywhere:
host = dbutils.notebook.entry_point.getDbutils().notebook().getContext().browserHostName().get()

[asnare]

Unclear what the issue is that this PR resolves, if anything it seems to prevent something that should fail from failing early.