debug: Looking into Azure Integration

[turbolytics]

refs #120

Debug

Validate Config

python cmd/sql-flow.py config validate dev/config/examples/azure-debug.yml

Invoke Dev Against Test Fixture to ensure that Pipeline SQL works

python cmd/sql-flow.py dev invoke dev/config/examples/azure-debug.yml dev/fixtures/azure.jsonl
2025-04-13 14:49:56,034 [INFO] executing command load extensions
2025-04-13 14:49:56,071 [INFO] executing command create adls secret
[{'payload': {'source': {'ts_ms': 1744569606922}}, 'SourceDatePartition': datetime.date(2025, 4, 13)}]

Debug DuckDB State while a pipeline is running

• Start SQLFlow with HTTP SQL Debug Enabled

python cmd/sql-flow.py run dev/config/examples/azure-debug.yml --with-http-debug
2025-04-13 15:00:42,662 [INFO] executing command load extensions
 * Serving Flask app 'sql-flow'
 * Debug mode: on
2025-04-13 15:00:42,696 [INFO] executing command create adls secret
2025-04-13 15:00:42,718 [INFO] consumer loop starting
2025-04-13 15:00:42,720 [INFO] WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
2025-04-13 15:00:42,720 [INFO] Press CTRL+C to quit

• Make a curl request to debug endpoint to list secrets

% curl 'localhost:5000/debug?sql=FROM%20duckdb_secrets%28%29%3B'
[
  [
    "adls_conn",
    "azure",
    "credential_chain",
    false,
    "memory",
    [
      "azure://",
      "az://",
      "abfss://",
      "abfs://"
    ],
    "name=adls_conn;type=azure;provider=credential_chain;serializable=true;scope=azure://,az://,abfss://,abfs://;account_name=;chain=default;cli;env"
  ]
]

The debug endpoint sql= param needs to be URL encoded. The command that is executed in this example is:

FROM duckdb_secrets();

https://duckdb.org/docs/stable/configuration/secrets_manager.html#listing-secrets