Enable Model Materialization as User Defined Function

[rfeith2]

Summary

Creates the ability to define the materialization type of a model as a user defined function in Dremio

Description

This PR introduces a new built-in udf materialization type for dbt-dremio that allows users to create user-defined functions (UDFs) directly from dbt models, addressing a key limitation where UDFs created as macros don't appear in the dependency graph (DAG).

Problem Statement:

• User Defined Functions are supported by both Dremio and dbt
• The default solution requires creating UDFs as dbt macros
• Critical Issue: Macros do not show up in the DAG, causing loss of dependency tracking and visibility in lineage graphs

Solution:
This implementation provides native UDF materialization support, ensuring UDFs are properly tracked in dbt's dependency graph and appear in data lineage visualizations.

Core Implementation:

• New UDF materialization macro (dbt/include/dremio/macros/materializations/udf.sql) that handles CREATE OR REPLACE FUNCTION statements
• Required configuration parameters: parameter_list and returns
• Optional database/schema targeting support (replaces the original folder_name approach)
• Proper relation handling and conflict resolution
• Full integration with dbt's dependency graph, enabling proper ref() usage and DAG visualization

Key Features:

• DAG Visibility: UDFs now appear in dependency graphs and lineage charts
• Dependency Tracking: Proper upstream/downstream relationship tracking
• Support for complex parameter types (INT, VARCHAR, DECIMAL, TABLE, etc.)
• Custom database and schema targeting via config
• Comprehensive validation with descriptive error messages
• CREATE OR REPLACE functionality for seamless model re-runs
• Integration with dbt's grant system and pre/post hooks

Usage Example:
{{ config(
materialized='udf',
parameter_list='start_date DATE, end_date DATE',
returns='TABLE(order_id INT, customer_name VARCHAR)'
) }}

RETURN
SELECT order_id, customer_name
FROM orders
WHERE order_date BETWEEN start_date AND end_date

Migration from Custom Implementation:
Users can now remove their custom UDF materialization macros and use this built-in implementation, gaining improved validation, error handling, and consistency with dbt-dremio conventions.

Test Results

Functional Tests (tests/functional/adapter/materialization/test_udf.py):

• Basic UDF creation and downstream model usage with proper DAG integration
• Complex parameter type variations including TABLE returns
• Custom database/schema configuration
• Error handling for missing required configs
• Nested UDF usage and dependency tracking verification
• Integration with dbt commands (compile, docs generate, ls)
• Downstream model referencing UDFs via ref() function

Unit Tests (tests/unit/test_udf_materialization.py):

• Macro file structure validation
• SQL generation correctness
• Jinja2 template syntax verification
• Configuration parameter handling

Hook Tests (tests/functional/adapter/materialization/test_udf_hooks.py):

• Pre/post hook execution with UDF materialization

Changelog

• [ Done] Added a summary of what this PR accomplishes to CHANGELOG.md

Contributor License Agreement

• [Done ] Please make sure you have signed our Contributor License Agreement, which enables Dremio to distribute your contribution without restriction.

Related Issue

N/A