[BUG]: Lakebridge analyzer fails to run on Windows machines that have corporate restrictions - e.g. LoadLibrary perl530.dll

[databricks-ostrowski]

Is there an existing issue for this?

• I have searched the existing issues

Category of Bug / Issue

Application crashed

Current Behavior

On enterprise machines, Lakebridge analyzer is having issues with some common Windows lockdowns. Analyzer should user more user-mode only approaches. This issue is for an ERROR and failure regarding the LoadLibrary perl530.dll

Expected Behavior

Lakebridge Analyzer completed.

Steps To Reproduce

PS C:\Work> databricks labs lakebridge analyze
Enter the path of the directory containing sources to analyze (default: C:/Work): C:/work/sql_scripts
Enter the path of the report file for analyzer results (default: lakebridge-analyzer-results.xlsx): C:/work/split_sql_scripts
Select the source technology:
[0]  ABInitio
[1]  ADF
[2]  Alteryx
[3]  Athena
[4]  BigQuery
[5]  BODS
[6]  Cloudera (Impala)
[7]  Datastage
[8]  Greenplum
[9]  Hive
[10] IBM DB2
[11] Informatica - Big Data Edition
[12] Informatica - PC
[13] Informatica Cloud
[14] Jupyter Notebook
[15] MS SQL Server
[16] Netezza
[17] Oozie
[18] Oracle
[19] Oracle Data Integrator
[20] PentahoDI
[21] PIG
[22] Presto
[23] PySpark
[24] Redshift
[25] SAPHANA - CalcViews
[26] SAS
[27] Snowflake
[28] SPSS
[29] SQOOP
[30] SSIS
[31] SSRS
[32] Synapse
[33] Talend
[34] Teradata
[35] Vertica
Enter a number between 0 and 35: 15

Relevant log output or Exception details

## Error / output

15:11:40 WARNING [src/databricks/labs/lakebridge/analyzer/lakebridge_analyzer] Excel report will be written without .xlsx extension: C:\work\split_sql_scripts
ERROR: LoadLibrary perl530.dll failed
15:11:46  ERROR [d.l.bladespector.analyzer] Analysis failed
Traceback (most recent call last):
  File "C:\work\.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\analyzer.py", line 153, in _run_binary
    raise subprocess.CalledProcessError(process.returncode, args)
subprocess.CalledProcessError: Command '['C:\\work\\.databricks\\labs\\lakebridge\\state\\venv\\Lib\\site-packages\\databricks\\labs\\bladespector\\Analyzer\\Windows\\analyzer.exe', '-d', 'C:\\work\\sql_scripts', '-r', 'C:\\Users\\CO16040.ARI_NT.000\\AppData\\Local\\Temp\\tmp6_0025ta\\staging-report.xlsx', '-t', 'SQL']' returned non-zero exit status 1.
15:11:47  ERROR [src/databricks/labs/lakebridge.analyze] RuntimeError: Analysis failed

Logs Confirmation

• I ran the command line with --debug
• I have attached the lsp-server.log under USER_HOME/.databricks/labs/remorph-transpilers/<converter_name>/lib/lsp-server.log

Sample Query

Operating System

Windows

Version

latest via Databricks CLI

[databricks-ostrowski]

Here is the --debug output:

databricks labs lakebridge analyze --debug
10:42:28 Info: start pid=66584 version=1.7.0 args="C:\Users\CO16040.ARI_NT.000\AppData\Local\Microsoft\WinGet\Packages\Databricks.DatabricksCLI_Microsoft.Winget.Source_8wekyb3d8bbwe\databricks.exe, labs, lakebridge, analyze, --debug"
10:42:28 Debug: Loading installed version info from: C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\version.json pid=66584
10:42:28 Debug: Loading login configuration from: C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\config\login.json pid=66584
10:42:28 Debug: Using workspace-level login profile: DEFAULT pid=66584
10:42:28 Debug: Loading DEFAULT profile from C:\Users\CO16040.ARI_NT.000.databrickscfg pid=66584 sdk=true
10:42:28 Debug: [Local Cache] using cache key: [REDACTED] pid=66584
10:42:28 Debug: [Local Cache] cache hit pid=66584
10:42:28 Debug: Resolved account_id from host metadata: "[REDACTED]" pid=66584 sdk=true
10:42:28 Debug: Resolved workspace_id from host metadata: "[REDACTED]" pid=66584 sdk=true
10:42:28 Debug: Resolved cloud from hostname: "Azure" pid=66584 sdk=true
10:42:28 Debug: Resolved token_audience from host metadata token_federation_default_oidc_audiences: "[REDACTED]" pid=66584 sdk=true
10:42:28 Debug: Resolved discovery_url from host metadata: "[REDACTED]" pid=66584 sdk=true
10:42:28 Debug: Resolved login: Config: host=[REDACTED].azuredatabricks.net, account_id=[REDACTED], workspace_id=[REDACTED], token=***, profile=DEFAULT, config_file=C:\Users\CO16040.ARI_NT.000.databrickscfg, databricks_cli_path=C:\Users\CO16040.ARI_NT.000\AppData\Local\Microsoft\WinGet\Packages\Databricks.DatabricksCLI_Microsoft.Winget.Source_8wekyb3d8bbwe\databricks.exe. Env: DATABRICKS_CLI_PATH pid=66584 sdk=true
10:42:28 Debug: Passing down environment variables: DATABRICKS_HOST, DATABRICKS_ACCOUNT_ID, DATABRICKS_TOKEN, DATABRICKS_CLI_PATH, DATABRICKS_CLOUD, DATABRICKS_DISCOVERY_URL, DATABRICKS_WORKSPACE_ID, DATABRICKS_AUTH_TYPE, DATABRICKS_TOKEN_AUDIENCE pid=66584
10:42:28 Debug: Forwarding subprocess: C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Scripts\python.exe C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\lib\src\databricks\labs\lakebridge\cli.py {"command":"analyze","flags":{"generate-json":"false","log_level":"debug","report-file":"","source-directory":"","source-tech":""},"output_type":""} pid=66584
10:42:28 Debug: starting: C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Scripts\python.exe C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\lib\src\databricks\labs\lakebridge\cli.py {"command":"analyze","flags":{"generate-json":"false","log_level":"debug","report-file":"","source-directory":"","source-tech":""},"output_type":""} pid=66584
10:42:35 DEBUG [databricks.sdk] Loaded from environment
10:42:35 DEBUG [databricks.sdk] Attempting to configure auth: pat
Enter the path of the directory containing sources to analyze (default: C:/Work): C:\Work\sql_scripts
Enter the path of the report file for analyzer results (default: lakebridge-analyzer-results.xlsx):
Select the source technology
[0] ABInitio
[1] ADF
[2] Alteryx
[3] Athena
[4] BigQuery
[5] BODS
[6] Cloudera (Impala)
[7] Datastage
[8] Greenplum
[9] Hive
[10] IBM DB2
[11] Informatica - Big Data Edition
[12] Informatica - PC
[13] Informatica Cloud
[14] Jupyter Notebook
[15] MS SQL Server
[16] Netezza
[17] Oozie
[18] Oracle
[19] Oracle Data Integrator
[20] PentahoDI
[21] PIG
[22] Presto
[23] PySpark
[24] Redshift
[25] SAPHANA - CalcViews
[26] SAS
[27] Snowflake
[28] SPSS
[29] SQOOP
[30] SSIS
[31] SSRS
[32] Synapse
[33] Talend
[34] Teradata
[35] Vertica
Enter a number between 0 and 35: 15
10:42:47 DEBUG [d.l.bladespector.analyzer] Using underlying analyzer: $C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\Analyzer\Windows\analyzer.exe
10:42:47 DEBUG [d.l.bladespector.analyzer] Running command: C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\Analyzer\Windows\analyzer.exe -d C:\Work\sql_scripts -r C:\Work\lakebridge-analyzer-results.xlsx -t SQL -v
ERROR: LoadLibrary perl530.dll failed
10:42:59 ERROR [d.l.bladespector.analyzer] Analysis failed
Traceback (most recent call last):
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\analyzer.py", line 153, in _run_binary
raise subprocess.CalledProcessError(process.returncode, args)
subprocess.CalledProcessError: Command '['C:\Users\CO16040.ARI_NT.000\.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\Analyzer\Windows\analyzer.exe', '-d', 'C:\Work\sql_scripts', '-r', 'C:\Work\lakebridge-analyzer-results.xlsx', '-t', 'SQL', '-v']' returned non-zero exit status 1.
10:42:59 DEBUG [d.l.l.contexts.application] Added User-Agent extra analyzer_error=RuntimeError
10:42:59 DEBUG [d.l.l.contexts.application] Added User-Agent extra cmd=analyze
10:43:00 DEBUG [databricks.sdk] GET /api/2.0/preview/scim/v2/Me
< 200 OK
< {
< "active": true,
< "displayName": "[REDACTED]",
< "emails": [
< {
< "primary": true,
< "type": "work",
< "value": "REDACTED"
< }
< ],
< "externalId": "[REDACTED]",
< "groups": [
< {
< "$ref": "[REDACTED]",
< "display": "[REDACTED]",
< "type": "direct",
< "value": "REDACTED"
< },
< "... (2 additional elements)"
< ],
< "id": "[REDACTED]",
< "name": {
< "familyName": "ID",
< "givenName": "[REDACTED]"
< },
< "schemas": [
< "urn:ietf:params:scim:schemas:core:2.0:User",
< "... (1 additional elements)"
< ],
< "userName": "[REDACTED]"
< }
10:43:00 ERROR [src/databricks/labs/lakebridge.analyze] Failed to call analyze
Traceback (most recent call last):
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\analyzer.py", line 153, in _run_binary
raise subprocess.CalledProcessError(process.returncode, args)
subprocess.CalledProcessError: Command '['C:\Users\CO16040.ARI_NT.000\.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\Analyzer\Windows\analyzer.exe', '-d', 'C:\Work\sql_scripts', '-r', 'C:\Work\lakebridge-analyzer-results.xlsx', '-t', 'SQL', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\blueprint\cli.py", line 120, in _route
cmd.fn(**kwargs)
~~~~~~^^^^^^^^^^
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\lib\src\databricks\labs\lakebridge\cli.py", line 946, in analyze
result = ctx.analyzer.run_analyzer(source_directory, report_file, source_tech, generate_json)
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\lib\src\databricks\labs\lakebridge\analyzer\lakebridge_analyzer.py", line 134, in run_analyzer
return self._runner.run(source_dir, results_file_path, platform, generate_json)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\lib\src\databricks\labs\lakebridge\analyzer\lakebridge_analyzer.py", line 91, in run
_runnable(source_dir, results_file_path, platform, self._is_debug, json_result)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\analyzer.py", line 106, in analyze
analyzer._run_binary(directory, result, technology, is_debug, json_result)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\CO16040.ARI_NT.000.databricks\labs\lakebridge\state\venv\Lib\site-packages\databricks\labs\bladespector\analyzer.py", line 162, in _run_binary
raise RuntimeError("Analysis failed") from e
RuntimeError: Analysis failed
10:43:01 Info: completed execution pid=66584 exit_code=0
10:43:01 Debug: no telemetry logs to upload pid=66584

[asnare]

@databricks-ostrowski: I'm not sure the analysis of this is correct, and based on #2600 I'd like you to confirm that this happens with TEMP/TMP not set.

Please also confirm what the Windows platform (x86-32, x86-64, arm, etc).