Support Databricks CLI authentication and early token refresh

[zacdav-db]

Summary

• route auth_type = databricks-cli through databricks auth token instead of the direct OAuth client
• keep provider selection bound to the selected .databrickscfg profile and follow the Databricks Python SDK profile precedence
• convert CLI responses into httr2_token credentials while leaving durable credentials and refresh to the CLI
• refresh OAuth tokens 40 seconds before expiry using the new httr2 1.3.0 expiry margin
• document the CLI authentication workflow and add offline coverage for routing, failures, expiry, cache reuse, and workspace isolation

Root cause

Brickster treated every tokenless request as a direct OAuth flow. Profiles created by databricks auth login identify databricks-cli as an external credential provider, so passing that value into Brickster's OAuth client could not reuse the CLI's secure credential storage. Older CLI versions appeared to work because they wrote a reusable token into the profile, which bypassed this tokenless OAuth path.

Databricks also rejects access tokens with 30 seconds or less remaining. Earlier httr2 releases could reuse those tokens because they only treated tokens as expired five seconds early. httr2 1.3.0 exposes an expiry margin through its normal refresh path, so Brickster now uses the same 40-second buffer as the Databricks Python SDK without manipulating cache internals.

Fixes #253.
Fixes #257.

Validation

• full devtools::test(reporter = "summary") suite with httr2 1.3.0 — passed; 21 expected live-workspace skips
• focused request/auth tests cover M2M, U2M, and CLI refresh with 35 seconds remaining
• Air formatting check on modified R code and tests
• git diff --check
• repository pre-commit and pre-push secret scans

[dabruehl]

@zacdav-db I tested the version. First attempt worked fine and I could authenticate and download tables.

However, closing R and retry it throws following error:

> con <- DBI::dbConnect(
+   drv = brickster::DatabricksSQL(),
+   warehouse_id = "<id>"
+ )
Error:
! Failed to connect to warehouse "<id>".
Caused by error in `value[[3L]]()`:
! The Databricks CLI returned a malformed OAuth token response.
Run `rlang::last_trace()` to see where the error occurred.

rlang::last_trace():

<error/rlang_error>
Error:
! Failed to connect to warehouse "<id>".
Caused by error in `value[[3L]]()`:
! The Databricks CLI returned a malformed OAuth token response.
---
Backtrace:
     ▆
  1. ├─DBI::dbConnect(drv = brickster::DatabricksSQL(), warehouse_id = "<id>")
  2. └─DBI::dbConnect(drv = brickster::DatabricksSQL(), warehouse_id = "<id>")
  3.   └─brickster (local) .local(drv, ...)
  4.     ├─base::tryCatch(...)
  5.     │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  6.     │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  7.     │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  8.     └─brickster::db_sql_query(...)
  9.       └─brickster:::db_sql_exec_and_wait(...)
 10.         └─brickster::db_sql_exec_query(...)
 11.           └─brickster:::db_perform_request(req)
 12.             ├─httr2::resp_body_json(...)
 13.             │ └─httr2:::check_response(resp)
 14.             │   └─httr2:::is_response(resp)
 15.             └─httr2::req_perform(httr2::req_error(req, body = db_req_error_body))
 16.               └─httr2:::req_perform1(...)
 17.                 └─httr2:::handle_preflight(req_prep, handle)
 18.                   └─httr2:::auth_sign(req)
 19.                     ├─rlang::exec(...)
 20.                     └─httr2 (local) `<fn>`(...)
 21.                       └─httr2:::auth_oauth_token_get(cache = cache, flow = flow, flow_params = flow_params)
 22.                         ├─rlang::exec(flow, !!!flow_params)
 23.                         └─brickster (local) `<fn>`(...)
 24.                           └─base::tryCatch(...)
 25.                             └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 26.                               └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 27.                                 └─value[[3L]](cond)

[zacdav-db]

@dabruehl I think that should now be fixed.

I think more broadly now that CLI managed auth exists in this branch I think it changes how I want to reason around the option to use use_databrickscfg, but that is for another time.

It may becomes confusing to end users, but its also important to allow users to continue to use .Renviron.

[dabruehl]

Hi @zacdav-db. I could test it and now it looks good. Thx