-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
2026-03-11T11:18:12.4937444Z 2-task-1-0/0 [default0]:tests/unit_tests/transformer/test_vision_cuda_graphs.py::TestVisionLayerIsGraphable::test_wrong_cuda_graph_impl_returns_false [default0]:FAILED[default0]:
2026-03-11T11:18:12.4941011Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4942176Z 2-task-1-0/0 [default0]:=================================== FAILURES ===================================
2026-03-11T11:18:12.4943134Z 2-task-1-0/0 [default0]:_____ TestVisionLayerIsGraphable.test_wrong_cuda_graph_impl_returns_false ______
2026-03-11T11:18:12.4943637Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4944148Z 2-task-1-0/0 [default0]:self = <tests.unit_tests.transformer.test_vision_cuda_graphs.TestVisionLayerIsGraphable object at 0x7669650ef0e0>
2026-03-11T11:18:12.4944696Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4945000Z 2-task-1-0/0 [default0]: def test_wrong_cuda_graph_impl_returns_false(self):
2026-03-11T11:18:12.4945519Z 2-task-1-0/0 [default0]: from megatron.core.transformer.transformer_layer import TransformerLayer
2026-03-11T11:18:12.4945933Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4946249Z 2-task-1-0/0 [default0]: config = SimpleNamespace(cuda_graph_impl="local")
2026-03-11T11:18:12.4946669Z 2-task-1-0/0 [default0]: layer = MagicMock(spec=TransformerLayer)
2026-03-11T11:18:12.4947089Z 2-task-1-0/0 [default0]: # isinstance check with MagicMock(spec=...) should pass
2026-03-11T11:18:12.4947789Z 2-task-1-0/0 [default0]:> assert _layer_is_graphable(layer, config) is False
2026-03-11T11:18:12.4948297Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4948600Z 2-task-1-0/0 [default0]:tests/unit_tests/transformer/test_vision_cuda_graphs.py:56:
2026-03-11T11:18:12.4949053Z 2-task-1-0/0 [default0]:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2026-03-11T11:18:12.4949394Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4949698Z 2-task-1-0/0 [default0]:layer = <MagicMock spec='TransformerLayer' id='130166166340384'>
2026-03-11T11:18:12.4950124Z 2-task-1-0/0 [default0]:config = namespace(cuda_graph_impl='local')
2026-03-11T11:18:12.4950432Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4950692Z 2-task-1-0/0 [default0]: def _layer_is_graphable(layer, config):
2026-03-11T11:18:12.4950990Z 2-task-1-0/0 [default0]: """
2026-03-11T11:18:12.4951255Z 2-task-1-0/0 [default0]: Check if a layer is graphable.
2026-03-11T11:18:12.4951540Z 2-task-1-0/0 [default0]: """
2026-03-11T11:18:12.4951783Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4952071Z 2-task-1-0/0 [default0]: # Only GraphableMegatronModule can be graphed.
2026-03-11T11:18:12.4952543Z 2-task-1-0/0 [default0]: if not isinstance(layer, GraphableMegatronModule):
2026-03-11T11:18:12.4952911Z 2-task-1-0/0 [default0]: return False
2026-03-11T11:18:12.4953152Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4953476Z 2-task-1-0/0 [default0]: # If cuda_graph_scope is not set, every layer is graphed.
2026-03-11T11:18:12.4953873Z 2-task-1-0/0 [default0]:> if not config.cuda_graph_scope:
2026-03-11T11:18:12.4954627Z 2-task-1-0/0 [default0]:E AttributeError: 'types.SimpleNamespace' object has no attribute 'cuda_graph_scope'. Did you mean: 'cuda_graph_impl'?
2026-03-11T11:18:12.4955176Z 2-task-1-0/0 [default0]:
2026-03-11T11:18:12.4955490Z 2-task-1-0/0 [default0]:megatron/core/transformer/cuda_graphs.py:1650: AttributeError
2026-03-11T11:18:12.4955950Z 2-task-1-0/0 [default0]:=============================== warnings summary ===============================
Steps/Code to reproduce bug
https://github.com/NVIDIA/Megatron-LM/actions/runs/22948551554/attempts/1
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working