feat: databricks deltalake dataset support

[akoumpa]

This PR adds Delta Lake / Databricks support for streaming instruction datasets in NeMo Automodel.

Key changes:

New DeltaLakeDataset delta_lake_dataset.py:694-732 - A streaming dataset that reads from Delta Lake tables (local, S3, Azure, GCS, or Databricks Unity Catalog). Automatically selects the best backend:

• deltalake library for simple tables
• Spark for tables with deletion vectors
• Databricks SQL Connector for Unity Catalog

New ColumnMappedTextInstructionIterableDataset column_mapped_text_instruction_iterable_dataset.py:108-119 - A streaming variant that accepts delta_storage_options, delta_sql_query parameters and routes Delta paths to the new backend.

Map-style dataset rejects Delta paths column_mapped_text_instruction_dataset.py:239-240 - Forces users to use the streaming variant for Delta Lake sources.

New ReservoirSampler reservoir_sampler.py:21-34 - Bounded-memory shuffle for streaming datasets.

Usage example (YAML):

dataset:
  _target_: ...IterableDataset
  path_or_dataset_id: delta://catalog
    .schema.training_data
  column_mapping:
    question: user_message
    answer: assistant_message
  delta_storage_options:
    DATABRICKS_TOKEN: ${oc.env:...}
    DATABRICKS_HOST: ${oc.env:...}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[floraxhuang]

Thanks for the PR! I tested this PR on a Databricks cluster and encountered a DeltaProtocolError. It appears the deltalake reader used here isn't compatible with tables that use Deletion Vectors or Column Mapping (which are now enabled by default on Databricks). Relevant discussions: delta-io/delta-rs#1094

• Databricks runtime: 15.4 LTS

• deltalake version: 1.2.1

• Error Trace:

DeltaProtocolError: The table has set these reader features: {'deletionVectors'} but these are not yet supported by the deltalake reader.

[akoumpa]

/ok to test 937cf25

[akoumpa]

/ok to test 7ca3800

[akoumpa]

/ok to test a460cb3

[akoumpa]

/ok to test 072146b

[HuiyingLi]

lgtm thanks!