Skip to content

Notebook: Databricks connector#128

Merged
andy-k-improving merged 15 commits intoft-hf-databricks-connector-feature-branchfrom
ft-hf-db-notebook
Apr 8, 2026
Merged

Notebook: Databricks connector#128
andy-k-improving merged 15 commits intoft-hf-databricks-connector-feature-branchfrom
ft-hf-db-notebook

Conversation

@andy-k-improving
Copy link
Copy Markdown
Contributor

@andy-k-improving andy-k-improving commented Apr 3, 2026

Summary

Adds a demo notebook for importing Databricks data into Neptune Analytics via Athena
Federation, along with a new get_athena_query_results utility function.

Changes

New: Databricks Import Demo Notebook

notebooks/import_databricks_demo.ipynb — End-to-end workflow demonstrating:

  • Downloading the Kaggle PaySim1 dataset via kagglehub
  • Uploading to Databricks Unity Catalog Volume via the Databricks SDK
  • Creating a Delta table using CREATE TABLE AS SELECT
  • Federating queries from Athena to Databricks via the athena-databricks-connector
  • Generating vertex/edge CSV projections for Neptune Analytics import
  • Running Louvain community detection on the imported graph

New: get_athena_query_results()

nx_neptune/instance_management.py — Fetches results for a completed Athena query by execution
ID. Paginates automatically. Returns raw rows (header + data) as list[list[str]].

Exported as public API in nx_neptune/init.py.

Tests

4 new tests for get_athena_query_results: single page, multi-page pagination, empty results,
default client creation.

@andy-k-improving andy-k-improving marked this pull request as ready for review April 3, 2026 01:46
mykola-pereyma
mykola-pereyma approved these changes Apr 6, 2026
View reviewed changes
nx_neptune/instance_management.py
)


def get_athena_query_results(
Copy link
Copy Markdown
Collaborator

@mykola-pereyma mykola-pereyma Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@acarbonetto @andy-k-improving - we may need to release minor version to support Databricks notebook

Copy link
Copy Markdown
Contributor

@acarbonetto acarbonetto Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. 0.6.0 or something.
We can ask if this requires that we go through AppSec. I'm hoping we can skip the appsec review this time.

acarbonetto
acarbonetto approved these changes Apr 7, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@acarbonetto acarbonetto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small comments. Overall, the notebook looks good

andy-k-improving and others added 15 commits April 8, 2026 12:29
@andy-k-improving
Init commit
c59e4b3 
Java module

Move src

Test file

minimise dep

Minimise dep

Update changes

Import driver

Update code

Update connection string

Update header
@andy-k-improving
Folder restrcuture
4db6878 
# Conflicts:
#	.gitmodules
#	connectors/athena-databricks-connector/pom.xml
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksCompositeHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksEnvironmentProperties.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksMetadataHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksRecordHandler.java
@andy-k-improving
Update tests
8e01c93
@andy-k-improving
SAM template
b1f33e3 
# Conflicts:
#	connectors/athena-databricks-connector/athena-databricks-connector.yaml
@andy-k-improving
Update doc
eae83a1
@andy-k-improving
Databricks notebook
02436a3 
Update notebook

Update notebook

Update dep

Update notebook
@andy-k-improving
Update notebook
e35bcef
@andy-k-improving
Update testcase
c38bb5e
@andy-k-improving @acarbonetto
Update notebooks/import_databricks_demo.ipynb
9826d7d 
Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
@andy-k-improving @acarbonetto
Update notebooks/import_databricks_demo.ipynb
dad45d9 
Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
@andy-k-improving @acarbonetto
Update notebooks/import_databricks_demo.ipynb
aad7639 
Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
@andy-k-improving @acarbonetto
Update notebooks/import_databricks_demo.ipynb
378fe15 
Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
@andy-k-improving @acarbonetto
Update notebooks/import_databricks_demo.ipynb
4d7df12 
Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
@andy-k-improving
Update doc
d411d8a
@andy-k-improving
Make NEV optional
01dfdb9
@andy-k-improving andy-k-improving changed the base branch from ft-hf-db-record-handler to ft-hf-databricks-connector-feature-branch April 8, 2026 19:31
@andy-k-improving andy-k-improving merged commit 090e247 into ft-hf-databricks-connector-feature-branch Apr 8, 2026
@andy-k-improving andy-k-improving deleted the ft-hf-db-notebook branch April 8, 2026 23:18
andy-k-improving added a commit that referenced this pull request Apr 9, 2026
@andy-k-improving @acarbonetto
Notebook: Databricks connector (#128)
ce7d133 
* Init commit

Java module

Move src

Test file

minimise dep

Minimise dep

Update changes

Import driver

Update code

Update connection string

Update header

* Folder restrcuture

# Conflicts:
#	.gitmodules
#	connectors/athena-databricks-connector/pom.xml
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksCompositeHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksEnvironmentProperties.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksMetadataHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksRecordHandler.java

* Update tests

* SAM template

# Conflicts:
#	connectors/athena-databricks-connector/athena-databricks-connector.yaml

* Update doc

* Databricks notebook

Update notebook

Update notebook

Update dep

Update notebook

* Update notebook

* Update testcase

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update doc

* Make NEV optional

---------

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
# Conflicts:
#	pyproject.toml
andy-k-improving added a commit that referenced this pull request Apr 10, 2026
@andy-k-improving @acarbonetto @mykola-pereyma
FEAT: Databricks connector (#133)
3f8aaba 
* Databricks connector - Module skeleton (#106)

* Init commit

Java module

Move src

Test file

minimise dep

Minimise dep

Update changes

Import driver

Update code

Update connection string

Update header

* Folder restrcuture

* Update tests

* Update Java doc

* POM refactor

* Update Readme

* Databricks Connector - Deployment Instruction (#107)

* SAM template

# Conflicts:
#	connectors/athena-databricks-connector/athena-databricks-connector.yaml

* Update template

* Update docker file

* Update doc

* Update doc

* Update doc

* Update doc

* Update doc

* Databricks Connector - Metadatahandler (#108)

* Update jdbc

* Ignore git module changes

Remove un compatible query

Update API signature

Update metadataHandler

Update metdata

Update wording

* Update doc

* Databricks connector - RecordHandler impelementation (#110)

* Init commit

Java module

Move src

Test file

minimise dep

Minimise dep

Update changes

Import driver

Update code

Update connection string

Update header

* Folder restrcuture

# Conflicts:
#	.gitmodules
#	connectors/athena-databricks-connector/pom.xml
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksCompositeHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksEnvironmentProperties.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksMetadataHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksRecordHandler.java

* Update tests

* SAM template

# Conflicts:
#	connectors/athena-databricks-connector/athena-databricks-connector.yaml

* Update doc

* Impl - Record handler

Working copy

Testable contructor

Unit test

Update record

Make fetch size configable

Update fetch size

Update doc

Update pushdown support

Unit test

* Update connectors/athena-databricks-connector/README.md

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update doc

# Conflicts:
#	connectors/athena-databricks-connector/README.md

---------

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Notebook: Databricks connector (#128)

* Init commit

Java module

Move src

Test file

minimise dep

Minimise dep

Update changes

Import driver

Update code

Update connection string

Update header

* Folder restrcuture

# Conflicts:
#	.gitmodules
#	connectors/athena-databricks-connector/pom.xml
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksCompositeHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksEnvironmentProperties.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksMetadataHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksRecordHandler.java

* Update tests

* SAM template

# Conflicts:
#	connectors/athena-databricks-connector/athena-databricks-connector.yaml

* Update doc

* Databricks notebook

Update notebook

Update notebook

Update dep

Update notebook

* Update notebook

* Update testcase

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update notebooks/import_databricks_demo.ipynb

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>

* Update doc

* Make NEV optional

---------

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
# Conflicts:
#	pyproject.toml

* Databricks connector: Deployment instruction fixes (#130)

* Init commit

Java module

Move src

Test file

minimise dep

Minimise dep

Update changes

Import driver

Update code

Update connection string

Update header

* Folder restrcuture

# Conflicts:
#	.gitmodules
#	connectors/athena-databricks-connector/pom.xml
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksCompositeHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksEnvironmentProperties.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksMetadataHandler.java
#	connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksRecordHandler.java

* Update tests

* SAM template

# Conflicts:
#	connectors/athena-databricks-connector/athena-databricks-connector.yaml

* Update doc

* Databricks notebook

Update notebook

Update notebook

Update dep

Update notebook

* Update testcase

* Update jdbc string

Update default option

Update log stmt

Reorder parameters

Minimise diff

* Update example host name

* Update notebook

* Connector: Directory restructure (#132)

* Move s3 vector

* Update path

* Misc update

* Update pyproject.toml

Co-authored-by: mykola-pereyma <pereymam@amazon.com>

* Update connectors/athena-databricks-connector/src/main/resources/log4j2.xml

Co-authored-by: mykola-pereyma <pereymam@amazon.com>

* Update connectors/athena-databricks-connector/Dockerfile

Co-authored-by: mykola-pereyma <pereymam@amazon.com>

* Update connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksCompositeHandler.java

Co-authored-by: mykola-pereyma <pereymam@amazon.com>

* Update connectors/athena-databricks-connector/athena-databricks-connector.yaml

Co-authored-by: mykola-pereyma <pereymam@amazon.com>

* Update connectors/athena-databricks-connector/README.md

Co-authored-by: mykola-pereyma <pereymam@amazon.com>

* Update connectors/athena-databricks-connector/src/main/java/com/amazonaws/athena/connectors/databricks/DatabricksRecordHandler.java

Co-authored-by: mykola-pereyma <pereymam@amazon.com>

* Fix notebook

* Update github action

* Update doc

* Update python

* Update java

* Update py test

---------

Co-authored-by: Andrew Carbonetto <andrew.carbonetto@improving.com>
Co-authored-by: mykola-pereyma <pereymam@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mykola-pereyma mykola-pereyma mykola-pereyma approved these changes

@acarbonetto acarbonetto acarbonetto approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@andy-k-improving @mykola-pereyma @acarbonetto