Unexperimentalize operators #6134

szalpal · 2025-12-16T16:06:39Z

Category:

Refactoring (Redesign of existing code that doesn't affect functionality)

Description:

This PR takes out of experimental the following operators:
experimental.audio_resample
experimental.debayer
experimental.equalize
experimental.filter
experimental.tensor_resize
nvidia.dali.plugin.numba.fn.experimental.numba_function

The experimental flavours of the operators is kept to maintain backwards compatibility.

Tests are configured to test both: experimental and regular flavours of the operators. Since the backwards compatibility is maintained, both of these operators need to be tested.

TODO:

Adjust documentation (hide experimental flavours)
Unexperimentalize decoders.video

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: DALI-4504

dali/test/python/type_annotations/test_typing_pipelines.py

dali/test/python/test_utils.py


-def check_numba_compatibility_gpu(if_skip=True):
-    import nvidia.dali.plugin.numba.experimental as ex
+def check_numba_compatibility_gpu(if_skip=True, use_experimental: bool = False):


szalpal · 2025-12-16T16:18:43Z

!build

dali-automaton · 2025-12-16T16:21:36Z

CI MESSAGE: [40306038]: BUILD STARTED

dali-automaton · 2025-12-16T21:24:51Z

CI MESSAGE: [40306038]: BUILD FAILED

dali/python/nvidia/dali/plugin/numba/__init__.py

+            # try importing cuda.core as it can be used later to check the compatibility
+            # it is okay to fail as it may not be installed, the check later can handle this
+            import cuda.core
+        except ImportError:


szalpal · 2025-12-17T07:16:22Z

!build

dali-automaton · 2025-12-17T07:20:58Z

CI MESSAGE: [40349534]: BUILD STARTED

szalpal · 2025-12-17T07:41:31Z

!build

dali-automaton · 2025-12-17T07:45:38Z

CI MESSAGE: [40350589]: BUILD STARTED

greptile-apps · 2025-12-17T11:15:13Z

Greptile Summary

This PR moves six stable DALI operators (audio_resample, debayer, equalize, filter, tensor_resize, and numba_function) out of the experimental namespace to regular operator status while maintaining backward compatibility through deprecated aliases
Test coverage has been extensively updated across multiple test files to validate both experimental and regular versions of all promoted operators using parameterized testing patterns
The decoders.video operator was also unexperimentalized as part of this effort, completing the promotion of operators that have proven stable through testing

Important Files Changed

Filename	Overview
dali/operators/image/convolution/filter.cc	Moves Filter operator schema from experimental to regular namespace with proper deprecation warnings and dual registration
dali/python/nvidia/dali/plugin/numba/init.py	Relocates complete NumbaFunction implementation from experimental to main module with 602 lines of moved code
dali/test/python/test_dali_variable_batch_size.py	Updates comprehensive variable batch size testing to cover both experimental and regular operator versions
dali/test/python/checkpointing/test_dali_stateless_operators.py	Refactors checkpointing tests to validate both operator namespaces for all affected operators
dali/operators/video/decoder/video_decoder_cpu.cc	Restructures video decoder schema from experimental to regular with inherited deprecation alias

Confidence score: 4/5

This refactoring follows established DALI patterns for operator graduation and maintains strict backward compatibility through dual registrations
Score reflects the comprehensive scope of changes across many files that require careful coordination, though implementation patterns are well-established
Pay close attention to test files to ensure parameterized testing correctly covers both experimental and regular operator variants

Sequence Diagram

sequenceDiagram
    participant User
    participant Pipeline
    participant OperatorRegistry
    participant Schema
    participant OperatorImpl

    User->>Pipeline: "fn.debayer(image, blue_position=[0,0])"
    Pipeline->>OperatorRegistry: "Lookup 'Debayer' operator"
    OperatorRegistry->>Schema: "Get DALI_SCHEMA(Debayer)"
    Schema-->>OperatorRegistry: "Return schema with args"
    OperatorRegistry->>OperatorImpl: "Create DebayerCPU/DebayerGPU instance"
    OperatorImpl-->>Pipeline: "Return operator instance"
    Pipeline->>OperatorImpl: "SetupImpl()"
    OperatorImpl-->>Pipeline: "Return output descriptor"
    Pipeline->>OperatorImpl: "RunImpl()"
    OperatorImpl-->>Pipeline: "Return processed output"
    Pipeline-->>User: "Return debayered image"

greptile-apps

Additional Comments (6)

dali/operators/image/color/equalize.cc, line 37 (link)

style: The docstring reference should use :meth:nvidia.dali.fn.equalize`` for consistency with DALI's documentation standards

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}
dali/python/nvidia/dali/plugin/numba/__init__.pyi, line 35 (link)

logic: Return type annotation may be incorrect - setup_fn appears to return None but the signature suggests it takes parameters and returns None

Should the setup_fn callable return type be None or should it return something else based on the actual implementation?
dali/python/nvidia/dali/plugin/numba/fn/__init__.pyi, line 36 (link)

syntax: The setup_fn parameter type annotation appears malformed - it has None as a return type in the callable signature but should likely return something or be Optional[Callable[...]] without the trailing None
dali/operators/generic/resize/tensor_resize_cpu.cc, line 49-62 (link)

style: The deprecated schema definition duplicates parent metadata (NumInput, NumOutput, SupportVolumetric, AllowSequences) that is already inherited from "TensorResize". Since AddParent("TensorResize") inherits all properties, these redundant specifications are unnecessary.

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}
dali/test/python/checkpointing/test_dali_stateless_operators.py, line 643-646 (link)

logic: inconsistent with other tests - this test only uses fn.audio_resample but the decorator includes both experimental and non-experimental versions
dali/python/nvidia/dali/plugin/numba/__init__.py, line 187 (link)

logic: Incorrect variable used: should be num_ins instead of num_outs for the input shapes calculation

_{25 files reviewed, 6 comments}

_{Edit Code Review Agent Settings | Greptile}

szalpal · 2025-12-17T17:05:14Z

!build

dali-automaton · 2025-12-17T17:11:34Z

CI MESSAGE: [40376844]: BUILD STARTED

dali-automaton · 2025-12-17T17:45:46Z

CI MESSAGE: [40376844]: BUILD FAILED

dali-automaton · 2025-12-17T20:58:13Z

CI MESSAGE: [40349534]: BUILD FAILED

szalpal · 2025-12-17T23:20:12Z

!build

dali-automaton · 2025-12-17T23:25:26Z

CI MESSAGE: [40397505]: BUILD STARTED

Signed-off-by: Michał Szołucha <[email protected]>

szalpal · 2026-01-13T18:09:38Z

!build

greptile-apps

_{1 file reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-01-13T18:11:37Z

Additional Comments (1)

dali/operators/image/color/debayer.cc
documentation example still references fn.experimental.debayer - should be updated to fn.debayer to demonstrate the stable API

    debayered_sequences = fn.debayer(
      bayered_sequences.gpu(), blue_position=blue_positions, algorithm='default_npp')

dali-automaton · 2026-01-13T18:16:10Z

CI MESSAGE: [41659598]: BUILD STARTED

dali-automaton · 2026-01-14T01:09:09Z

CI MESSAGE: [41659598]: BUILD FAILED

szalpal · 2026-01-14T10:52:47Z

!build

dali-automaton · 2026-01-14T10:55:32Z

CI MESSAGE: [41717403]: BUILD STARTED

greptile-apps

_{29 files reviewed, 15 comments}

_{Edit Code Review Agent Settings | Greptile}

dali/test/python/operator_2/test_filter.py

Signed-off-by: Michał Szołucha <[email protected]>

szalpal · 2026-01-14T11:10:58Z

!build

dali-automaton · 2026-01-14T11:16:35Z

CI MESSAGE: [41719186]: BUILD STARTED

greptile-apps

Additional Comments (1)

dali/operators/image/color/debayer.cc, line 51 (link)

style: Example in documentation still uses experimental namespace and should be updated to use fn.debayer

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

_{29 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

greptile-apps · 2026-01-14T11:26:42Z

dali/python/nvidia/dali/plugin/numba/fn/__init__.pyi

+    in_types: List[DALIDataType],
+    outs_ndim: List[int],
+    ins_ndim: List[int],
+    setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]], None]]] = None,


logic: The setup_fn type annotation has an invalid callable signature with None as the third parameter type. Callable type parameters should only specify input argument types, not return types. Should be Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], None]] if it returns None, or just Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], Any]].

Suggested change

setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]], None]]] = None,

setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], None]] = None,

greptile-apps · 2026-01-14T11:26:49Z

dali/python/nvidia/dali/plugin/numba/__init__.py

+                in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_outs, num_samples)
+                setup_fn(out_shapes_np, in_shapes_np)


logic: Bug in parameter usage: should be num_ins instead of num_outs for input shapes.

Suggested change

in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_outs, num_samples)

setup_fn(out_shapes_np, in_shapes_np)

in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_ins, num_samples)

setup_fn(out_shapes_np, in_shapes_np)

dali-automaton · 2026-01-14T20:10:13Z

CI MESSAGE: [41717403]: BUILD FAILED

dali-automaton · 2026-01-15T00:43:33Z

CI MESSAGE: [41719186]: BUILD FAILED

github-advanced-security bot found potential problems Dec 16, 2025

View reviewed changes

szalpal force-pushed the unexperimentalize branch from d372995 to 5aa731e Compare December 16, 2025 20:20

szalpal force-pushed the unexperimentalize branch 3 times, most recently from e630529 to 51e599d Compare December 17, 2025 07:12

github-advanced-security bot found potential problems Dec 17, 2025

View reviewed changes

szalpal force-pushed the unexperimentalize branch 2 times, most recently from a3c3ac6 to a424878 Compare December 17, 2025 07:41

szalpal marked this pull request as ready for review December 17, 2025 11:06

greptile-apps bot reviewed Dec 17, 2025

View reviewed changes

dali-automaton assigned szkarpinski and stiepan Dec 17, 2025

szalpal marked this pull request as draft December 17, 2025 13:13

szalpal force-pushed the unexperimentalize branch from a424878 to d5291dc Compare December 17, 2025 17:05

szalpal force-pushed the unexperimentalize branch from d5291dc to e1c19f9 Compare December 17, 2025 23:19

szalpal force-pushed the unexperimentalize branch from e1c19f9 to 0d5e199 Compare December 17, 2025 23:37

szkarpinski approved these changes Dec 22, 2025

View reviewed changes

szalpal added 9 commits January 13, 2026 18:48

Unexperimentalize tensor_resize operator

1e76570

Signed-off-by: Michał Szołucha <[email protected]>

Unexperimentalize debayer operator

41528dd

Signed-off-by: Michał Szołucha <[email protected]>

Unexperimentalize equalize operator

38cf1f4

Signed-off-by: Michał Szołucha <[email protected]>

Unexperimentalize filter operator

be0510a

Signed-off-by: Michał Szołucha <[email protected]>

Unexperimentalize numba_function

93ddead

Signed-off-by: Michał Szołucha <[email protected]>

Unexperimentalize video decoder

c34c213

Signed-off-by: Michał Szołucha <[email protected]>

Adjust numba function tests

a904b3f

Signed-off-by: Michał Szołucha <[email protected]>

Adjust debayer tests

5659252

Signed-off-by: Michał Szołucha <[email protected]>

Adjust equalize tests

c74140a

Signed-off-by: Michał Szołucha <[email protected]>

szalpal force-pushed the unexperimentalize branch from 8ffd54c to 27a2895 Compare January 13, 2026 18:08

greptile-apps bot reviewed Jan 13, 2026

View reviewed changes

szalpal force-pushed the unexperimentalize branch from 27a2895 to 93131c0 Compare January 14, 2026 10:51

greptile-apps bot reviewed Jan 14, 2026

View reviewed changes

szalpal added 4 commits January 14, 2026 12:09

Adjust Filter tests

c2795d2

Signed-off-by: Michał Szołucha <[email protected]>

Adjust video decoder tests

58fc81f

Signed-off-by: Michał Szołucha <[email protected]>

Adjust stateless tests

5fc0aef

Signed-off-by: Michał Szołucha <[email protected]>

Adjust CPU only and variable batch size tests

114a2cf

Signed-off-by: Michał Szołucha <[email protected]>

szalpal force-pushed the unexperimentalize branch from 93131c0 to 114a2cf Compare January 14, 2026 11:09

greptile-apps bot reviewed Jan 14, 2026

View reviewed changes

	setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]], None]]] = None,
	setup_fn: Optional[Callable[[Sequence[Sequence[Any]], Sequence[Sequence[Any]]], None]] = None,

		in_shapes_np = _get_shape_view(in_shapes_ptr, in_ndims_ptr, num_outs, num_samples)
		setup_fn(out_shapes_np, in_shapes_np)

Unexperimentalize operators #6134

Are you sure you want to change the base?

Unexperimentalize operators #6134

Uh oh!

Conversation

szalpal commented Dec 16, 2025 • edited by JanuszL Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

Uh oh!

Uh oh!

Check notice

szalpal commented Dec 16, 2025

Uh oh!

dali-automaton commented Dec 16, 2025

Uh oh!

dali-automaton commented Dec 16, 2025

Uh oh!

Check notice

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

greptile-apps bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Important Files Changed

Confidence score: 4/5

Sequence Diagram

Uh oh!

greptile-apps bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Additional Comments (6)

Uh oh!

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

szalpal commented Dec 17, 2025

Uh oh!

dali-automaton commented Dec 17, 2025

Uh oh!

szalpal commented Jan 13, 2026

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot commented Jan 13, 2026

Uh oh!

dali-automaton commented Jan 13, 2026

Uh oh!

dali-automaton commented Jan 14, 2026

Uh oh!

szalpal commented Jan 14, 2026

Uh oh!

dali-automaton commented Jan 14, 2026

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

szalpal commented Dec 16, 2025 •

edited by JanuszL

Loading

greptile-apps bot commented Dec 17, 2025 •

edited

Loading

greptile-apps bot left a comment •

edited

Loading

greptile-apps bot left a comment •

edited

Loading