[BUG][SearchStats] Request took time stats is not functioning correctly

[sgup432]

Describe the bug

We introduced changes sometime back to track coordinator level stats for search here.

But it seems like it doesn't work correctly like in below example(taken from a OpenSearch cluster):

    "took": {
        "time_in_millis": 53948,
        "current": 9353,
        "total": 51511
    },
    "dfs_pre_query": {
        "time_in_millis": 0,
        "current": 0,
        "total": 0
    },
    "query": {
        "time_in_millis": 44618,
        "current": 0,
        "total": 13866
    },
    "fetch": {
        "time_in_millis": 5413,
        "current": 0,
        "total": 13866
    },
    "dfs_query": {
        "time_in_millis": 0,
        "current": 0,
        "total": 0
    },
    "expand": {
        "time_in_millis": 15,
        "current": 0,
        "total": 13866
    },
    "can_match": {
        "time_in_millis": 0,
        "current": 0,
        "total": 0
    }

Here, we can see that the current metric for took is incorrect and fails to decrement in many scenarios. Also it should be equal to sum of each phase level current counter. Similarly time_in_millis and total also might be incorrect.

There appears to be a bug in our request stats logic where the counter is incremented but not decremented properly. The code path responsible for decrementing it is never executed, and we need to handle these scenarios.

Related component

No response

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

• current counter metric for took time should decrement correctly in all scenarios.
• current metric should ideally be equal to sum of each phase level current counter metric

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[sgup432]

@dzane17 Can you look into this?