Add llm-transpile command with Switch runner and integration tests

[hiroyukinakazato-db]

Changes

This PR adds the llm-transpile command for LLM-powered SQL conversion using the Switch transpiler.

What does this PR do?

Adds llm-transpile CLI command that runs Switch transpiler jobs with parameter passing support.

Relevant implementation details

CLI Integration:

• Add llm-transpile command to Lakebridge CLI
• Input source validation (workspace paths and local files)
• Parameter passing to Switch job runs

Switch Runner Implementation:

• SwitchConfig: manages Switch resources and job ID retrieval from InstallState
• SwitchRunner: orchestrates Switch job execution with parameters

Testing:

• Unit tests for Switch runner with parameter verification
• Integration tests for Switch installation lifecycle

Development Environment:

• Add .env to .gitignore for local development credentials

Caveats/things to watch out for when reviewing:

• Parameter design: Follows transpile and recon command patterns
• Catalog/schema usage: Uses values configured during Switch installation (following recon pattern)
• Output parameter naming: Uses --output-ws-folder (not --output-folder) to explicitly indicate workspace folder
• Dependencies: Requires PR Add Switch transpiler with --include-llm-transpiler flag #2066 (Switch installation) to be merged first

~ ❯ databricks labs lakebridge llm-transpile --input-source $HOME/IdeaProjects/switch/examples/workflow/airflow/input --output-ws-folder /Workspace/Users/<>/transpiled --source-dialect airflow

17:23:54     INFO [d.labs.lakebridge] Please read and accept the following comments before proceeding:
        This Feature leverages a large language model (LLM) to analyse and convert your provided content, code and data.
        You consent to your content being transmitted to, processed by, and returned from the LLM hosted by Databricks foundational models or other external models you may configure during the runtime.
        The outputs of the LLM are generated automatically without human review, and may contain inaccuracies or errors.
        You are responsible for reviewing and validating all outputs before relying on them for any critical or production use.
        By running this feature you accept these conditions.

Enter catalog name (default: lakebridge): lakebridge
17:24:11     INFO [d.l.l.deployment.configurator] Found existing catalog `lakebridge`
Enter schema name (default: switch):
17:24:15     INFO [d.l.l.deployment.configurator] Found existing schema `switch` in catalog `lakebridge`
Enter volume name (default: switch_volume):
17:24:18     INFO [d.l.l.deployment.configurator] Found existing volume `switch_volume` in catalog `lakebridge` and schema `switch`
Select a Foundation Model serving endpoint:
[0] [Recommended] databricks-claude-sonnet-4-5
[1] databricks-bge-large-en
[2] databricks-claude-3-7-sonnet
[3] databricks-claude-opus-4
[4] databricks-claude-opus-4-1
[5] databricks-claude-sonnet-4
[6] databricks-gemini-2-5-flash
[7] databricks-gemini-2-5-pro
[8] databricks-gemma-3-12b
[9] databricks-gpt-5
[10] databricks-gpt-5-mini
[11] databricks-gpt-5-nano
[12] databricks-gpt-oss-120b
[13] databricks-gpt-oss-20b
[14] databricks-gte-large-en
[15] databricks-llama-4-maverick
[16] databricks-meta-llama-3-1-405b-instruct
[17] databricks-meta-llama-3-1-8b-instruct
[18] databricks-meta-llama-3-3-70b-instruct
[19] databricks-qwen3-next-80b-a3b-instruct
[20] databricks-shutterstock-imageai
Enter a number between 0 and 20: 0
17:24:32     INFO [d.l.l.transpiler.switch_runner] Uploading /Users/<>/IdeaProjects/switch/examples/workflow/airflow/input to /Volumes/lakebridge/switch/switch_volume/input_20251105115432_n2iz...
17:24:33     INFO [d.l.l.transpiler.switch_runner] Upload complete: /Volumes/lakebridge/switch/switch_volume/input_20251105115432_n2iz
17:24:33     INFO [d.l.l.transpiler.switch_runner] Triggering Switch job with job_id: job_id
17:24:34     INFO [d.l.l.transpiler.switch_runner] Switch LLM transpilation job started: https://<workspacename>/jobs/job_id/runs/run_id
[
  {
    "job_id": job_id,
    "run_id": run_id,
    "run_url": "https://<workspacename/jobs/job_id/runs/run_id"
  }
]%

Linked issues

Resolves #2047

Functionality

• added relevant user documentation
• added new CLI command: databricks labs lakebridge llm-transpile
• modified existing command

Tests

• manually tested
• added unit tests
• added integration tests

[github-actions]

✅ 51/51 passed, 9 flaky, 3m31s total

Flaky tests:

• 🤪 test_validate_mixed_checks (160ms)
• 🤪 test_validate_invalid_schema_path (1ms)
• 🤪 test_validate_successful_schema_check (184ms)
• 🤪 test_validate_non_empty_tables (11ms)
• 🤪 test_validate_invalid_schema_check (1ms)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (13.936s)
• 🤪 test_transpiles_informatica_to_sparksql (15.902s)
• 🤪 test_transpile_teradata_sql_non_interactive[True] (17.204s)
• 🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (4.268s)

_{Running from acceptance #2885}

[asnare]

One thing we need to do is make sure we're consistently using the term foundation model, not foundational model.

[asnare]

Is this file intentionally blank?

[sundarshankar89]

I intended to remove this, since tests for test_cli_llm_transpile.py cover tests for switch_runner.py

[asnare]

Hmm, I'm not sure what this is for?

[sundarshankar89]

In case there is an error while running the job!!

[asnare]

We should skip files and directories that start with a period (.).

[sundarshankar89]

Added a small logic for that