Standardize Profiler Path Extract

[sundarshankar89]

Changes

What does this PR do?

Standardize Profiler Path Extract

Relevant implementation details

Caveats/things to watch out for when reviewing:

Linked issues

Resolves #..

Functionality

• added relevant user documentation
• added new CLI command
• modified existing command: databricks labs lakebridge ...
• ... +add your own

Tests

• manually tested
• added unit tests
• added integration tests

[codecov]

Codecov Report

❌ Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.95%. Comparing base (5edc2b5) to head (3565d4d).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...databricks/labs/lakebridge/assessments/pipeline.py	0.00%	1 Missing ⚠️
.../labs/lakebridge/assessments/profiler_validator.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2237   +/-   ##
=======================================
  Coverage   63.95%   63.95%           
=======================================
  Files          99       99           
  Lines        8644     8644           
  Branches      890      890           
=======================================
  Hits         5528     5528           
  Misses       2944     2944           
  Partials      172      172           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 130/130 passed, 8 flaky, 5 skipped, 9m32s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (21.851s)
• 🤪 test_installs_and_runs_pypi_bladebridge (27.102s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (17.705s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (18.264s)
• 🤪 test_transpile_teradata_sql (19.233s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (3.856s)
• 🤪 test_recon_databricks_job_succeeds (25.346s)
• 🤪 test_transpile_teradata_sql_non_interactive[False] (13.445s)

_{Running from acceptance #3486}

[goodwillpunning]

FYI we'll need to add the run Id as a path prefix in a follow-up PR.

[sundarshankar89]

why at the folder level? can we not do it at table level inside the database?

[gueniai]

Why don't we make the profiler output configurable by the user?

[sundarshankar89]

having it central will help us share state more easily across component, That was the principle behind a common central location.

[gueniai]

I can see the value of that. Maybe we need to think about how to manage the state for all modules and then standardize the optionality of outputs for all.

[goodwillpunning]

LGTM!

[eri-adepoju]

LGTM