Enrich failure handling

[Rei1010]

What type of PR is this?

/kind feature

What this PR does / why we need it:
Improving failure handling for test.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Error logs:

  [FAILED] in [It] - /Users/rui/Documents/Repos/Rei1010/HAMi/test/e2e/pod/test_pod.go:156 @ 05/21/25 11:09:07.958
  STEP: Check pod detailed after each test @ 05/21/25 11:09:07.958
I0521 11:09:07.977833   30711 pod.go:179] Pod default/gpu-pod5729 is in Pending status
I0521 11:09:07.977850   30711 pod.go:181] Show events for default/gpu-pod5729:
I0521 11:09:07.986884   30711 pod.go:190] Reason: FailedScheduling, Message : 0/1 nodes are available: 1 Insufficient nvidia.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.. 
I0521 11:09:07.993636   30711 pod.go:200] Show logs for default/gpu-pod5729:
I0521 11:09:07.993648   30711 pod.go:201] 

Does this PR introduce a user-facing change?:

[codecov]

Codecov Report

❌ Patch coverage is 82.48175% with 24 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/device/ascend/device.go	82.48%	21 Missing and 3 partials ⚠️

Flag	Coverage Δ
unittests	61.07% <82.48%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
pkg/device/cambricon/device.go	79.71% <ø> (ø)
pkg/device/devices.go	73.65% <ø> (ø)
pkg/device/enflame/device.go	78.08% <ø> (ø)
pkg/device/hygon/device.go	93.37% <ø> (ø)
pkg/device/iluvatar/device.go	86.23% <ø> (ø)
pkg/device/metax/config.go	100.00% <ø> (ø)
pkg/device/metax/device.go	81.13% <ø> (ø)
pkg/device/metax/protocol.go	62.74% <ø> (ø)
pkg/device/metax/sdevice.go	21.51% <ø> (ø)
pkg/device/mthreads/device.go	90.90% <ø> (ø)
... and 23 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull Request Overview

This PR enhances test failure diagnostics by adding utilities to fetch and log pod details across namespaces, adjusts the pod‐running wait interval for GPU workloads, and integrates detailed pod checks after any test failure.

• Increased the polling interval in WaitForPodRunning from 5s to 30s.
• Introduced GetNamespaceList, GetPodLogs, and CheckPodDetails in test/utils/pod.go.
• Updated AfterEach in test/e2e/pod/test_pod.go to call CheckPodDetails on failures and removed a debug fmt.Printf.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
test/utils/pod.go	Added new failure-handling helpers, updated polling interval, and imported I/O packages.
test/e2e/pod/test_pod.go	Call CheckPodDetails on test failures and remove leftover fmt.Printf debug statement.

Comments suppressed due to low confidence (1)

test/utils/pod.go:96

• Use the passed-in context ctx instead of context.TODO() to allow cancellation and deadlines to propagate correctly.

pod, err := clientSet.CoreV1().Pods(namespace).Get(context.TODO(), podName, metav1.GetOptions{})

[Copilot]

Returning here stops logging details for other pods. Consider using continue to proceed to the next pod and log all failures.

Suggested change

	return
	continue

[Copilot]

As with events, use continue instead of return so that other pods are still checked and logged.

Suggested change

	return
	continue

[Copilot]

[nitpick] Passing raw logs to Infof can misinterpret formatting verbs—use klog.Info(logs) or klog.Infof("%s", logs) instead.

Suggested change

	klog.Infof(logs)
	klog.Infof("%s", logs)

[Copilot]

[nitpick] Add a doc comment to describe the purpose and behavior of this public function for better maintainability.

Suggested change

	// GetNamespaceList retrieves a list of all namespaces in the Kubernetes cluster.
	// It takes a Kubernetes clientset as input and returns a slice of namespace names
	// or an error if the operation fails.