Syscall Timeout for evtNext in Windows Event Log Receiver

[pankaj101A]

Component(s)

receiver/windowseventlog

Is your feature request related to a problem? Please describe.

The evtNext function in api.go wraps the Windows EvtNext syscall. The current caller in subscription.go invokes it with timeout=0.

Timeout=0 tells the Windows API "don't wait for new events, return immediately." However, this parameter only controls the event-wait phase. It does not bound the time spent in the execution phase — including RPC transport, kernel I/O, storage driver operations, and security descriptor checks. These layers can hang indefinitely regardless of the timeout value.

This issue becomes significantly more severe in enterprise deployments where the OpenTelemetry Collector is configured to auto-discover and collect from Active Directory Domain Controllers (DCs).

Domain Controllers are uniquely problematic for this issue due to high number of evtNext calls

Describe the solution you'd like

Add a Go-level context-aware timeout wrapper at the Subscription level in subscription.go, leaving api.go unchanged:

Something like Below

var (
    errSyscallTimeout           = errors.New("evtNext syscall safety timeout exceeded")
    defaultSyscallSafetyTimeout = 10 * time.Second
)

func (s *Subscription) evtNextWithTimeout(
    ctx context.Context,
    resultSet uintptr,
    eventsSize uint32,
    events *uintptr,
    timeout, flags uint32,
    returned *uint32,
) error {
    done := make(chan error, 1)
    go func() {
        done <- evtNext(resultSet, eventsSize, events, timeout, flags, returned)
    }()

    safetyTimeout := s.syscallSafetyTimeout
    if safetyTimeout == 0 {
        safetyTimeout = defaultSyscallSafetyTimeout
    }
    totalTimeout := time.Duration(timeout)*time.Millisecond + safetyTimeout

    timer := time.NewTimer(totalTimeout)
    defer timer.Stop()

    select {
    case err := <-done:
        return err
    case <-ctx.Done():
        return ctx.Err()
    case <-timer.C:
        s.orphanedGoroutines.Add(1)
        s.logger.Warn("evtNext did not return within safety timeout; goroutine orphaned",
            zap.Duration("safety_timeout", totalTimeout),
            zap.Int64("orphaned_goroutines", s.orphanedGoroutines.Load()),
            zap.String("channel", s.channel),
        )
        return errSyscallTimeout
    }
}

Describe alternatives you've considered

No response

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• receiver/windowseventlog: @armstrmi @pjanotti

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[pjanotti]

cc @mrsillydog that has being very active in the receiver and related code lately

@pankaj101A

It does not bound the time spent in the execution phase — including RPC transport, kernel I/O, storage driver operations, and security descriptor checks. These layers can hang indefinitely regardless of the timeout value.

This seems like a bug on the EvtNext API, anyway, do you have some more info on that? Reports or a repro?

[mrsillydog]

@pjanotti Thank you for tagging me on this!

One early concern with the approach outlined here - a Go-level context timeout does not cancel the actual syscall, which is interacting directly with local variables (eventHandles and eventsRead in (s *Subscription) readWithRetry). If the timeout occurs and an error is returned, the memory for those local variables will be reclaimed by GC, and potentially reallocated. If the orphaned syscall were to return after that, unsafe memory modification could occur.

Given the syscall is working directly with memory, we need to be extremely careful about what memory the orphaned Go routines/syscalls has access to.

+1 for reports or a repro.

[vyagh]

hi, should i work on this issue?

[pankaj101A]

@vyagh @pjanotti i will try to reproduce it first then we can take a call

[pankaj101A]

i am not able to reproduce the infinite hang situation on my local due as it required network partition and congestion. But its mentioned in this article by mircosoft the https://learn.microsoft.com/en-us/windows/win32/rpc/preventing-client-side-hangs rpc can hang for 12 mintues (not indefinite but still high)