Fix torch.compile breaking toggle_optimizer / untoggle_optimizer (#21… #30660

Job	Run time
build-packages / build-packages (fabric)	23s
build-packages / build-packages (pytorch)	22s
build-packages / build-packages (lightning)	31s
build-packages / merge-artifacts	12s
install-pkg (ubuntu-22.04, lightning, 3.11)	1m 42s
install-pkg (ubuntu-22.04, notset, 3.10)	1m 39s
install-pkg (ubuntu-22.04, pytorch, 3.11)	1m 43s
install-pkg (ubuntu-22.04, fabric, 3.10)	1m 31s
install-pkg (ubuntu-22.04, fabric, 3.11)	1m 33s
install-pkg (ubuntu-22.04, lightning, 3.10)	1m 33s
install-pkg (macOS-14, fabric, 3.11)	34s
install-pkg (ubuntu-22.04, pytorch, 3.10)	1m 35s
install-pkg (macOS-14, pytorch, 3.11)	36s
install-pkg (ubuntu-22.04, notset, 3.11)	1m 34s
install-pkg (macOS-14, notset, 3.10)	50s
install-pkg (windows-2022, fabric, 3.10)	1m 36s
install-pkg (windows-2022, pytorch, 3.11)	1m 58s
install-pkg (windows-2022, notset, 3.10)	1m 44s
install-pkg (windows-2022, lightning, 3.11)	2m 40s
install-pkg (macOS-14, fabric, 3.10)	57s
install-pkg (windows-2022, fabric, 3.11)	1m 40s
install-pkg (windows-2022, notset, 3.11)	2m 1s
install-pkg (macOS-14, lightning, 3.11)	50s
install-pkg (macOS-14, notset, 3.11)	54s
install-pkg (windows-2022, pytorch, 3.10)	2m 14s
install-pkg (windows-2022, lightning, 3.10)	1m 57s
install-pkg (macOS-14, pytorch, 3.10)	53s
install-pkg (macOS-14, lightning, 3.10)	48s
install-pkg-guardian	3s
	36m 33s

Provide feedback