Update the dependency installation process to capture and log output

[eri-adepoju]

Changes

• Updated the dependency installation process to capture output instead of redirecting to DEVNULL, improving error visibility.
• Enhanced error logging to provide clearer messages by stripping unnecessary whitespace from stderr.

Linked issues

Resolves #2225

Functionality

• modified the run command to use capture_output=True and text=True for better output handling.

Tests

• manually tested
• added unit tests
• added integration tests

[codecov]

Codecov Report

❌ Patch coverage is 0% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.01%. Comparing base (254b5d3) to head (5973698).

Files with missing lines	Patch %	Lines
...databricks/labs/lakebridge/assessments/pipeline.py	0.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2226      +/-   ##
==========================================
- Coverage   64.05%   64.01%   -0.04%     
==========================================
  Files         100      100              
  Lines        8624     8629       +5     
  Branches      893      895       +2     
==========================================
  Hits         5524     5524              
- Misses       2928     2933       +5     
  Partials      172      172              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ 129/129 passed, 9 flaky, 5 skipped, 10m22s total

Flaky tests:

• 🤪 test_installs_and_runs_local_bladebridge (20.499s)
• 🤪 test_test_recon_capture_start_databricks_row (5.665s)
• 🤪 test_recon_capture_start_oracle_with_exception (5.672s)
• 🤪 test_apply_threshold_for_only_threshold_mismatch_with_true_absolute (5.865s)
• 🤪 test_installs_and_runs_pypi_bladebridge (23.364s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (21.396s)
• 🤪 test_transpiles_informatica_to_sparksql (22.906s)
• 🤪 test_transpile_teradata_sql (22.163s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (5.066s)

_{Running from acceptance #3438}

[sundarshankar89]

@eri-adepoju, can this only be part of the debug log level, not during regular processing?

[eri-adepoju]

@sundarshankar89 I've edited it to be more verbose at the debug level.

[gueniai]

LGTM