Add % of resources usage in `nomad node status --verbose`

[asyd]

Proposal

I would like to have percentage of RAM/CPU/Disks usage by node when calling nomad node status --verbose

Use-cases

Improve efficiency when having resources exhausted error after a deployment

Attempted Solutions

[gulducat]

Heya @asyd , interesting suggestion!

Is this the information you're looking for?

$ nomad node status 907023aa | grep Allocated -A2
Allocated Resources
CPU          Memory      Disk
0/25400 MHz  0 B/31 GiB  0 B/15 GiB

I hesitate to add three more columns (but not strictly columnar, since the values have spaces) to the top-level every-node output, even with it being -verbose, since it's already a bit wide.

Glancing at your github profile, I suspect you may be already familiar with this, but this is reasonably easy to script with something like:

nomad node status | awk '$8 == "ready" {print $1}' \
| xargs -P1 -I{} sh -c 'echo === {} ===; nomad node status {} | grep -A2 Allocated | tail -n2'

or, with a loop instead of xargs,

nomad node status | awk '$8 == "ready" {print $1}' | while read -r id; do
  echo === $id ===
  nomad node status $id | grep -A2 Allocated | tail -n2
done

It is more API calls, but they're pretty lightweight since they just fetch info.

I might wrap that in a function() { } and put it in my ~/.bashrc (or similar) to run more easily.

Either way, the output looks like this:

=== f84ec21f ===
CPU          Memory      Disk
0/25400 MHz  0 B/31 GiB  0 B/776 GiB
=== b5575335 ===
CPU          Memory      Disk
0/25400 MHz  0 B/31 GiB  0 B/776 GiB
=== 92dc056b ===
CPU          Memory      Disk
0/25400 MHz  0 B/31 GiB  0 B/776 GiB
=== 7399d3e0 ===
CPU          Memory      Disk
0/25400 MHz  0 B/31 GiB  0 B/776 GiB
=== 907023aa ===
CPU            Memory          Disk
400/25400 MHz  1.2 GiB/31 GiB  1.2 GiB/15 GiB

Of course you can modify that however you might like to look nicer.

Would that suit your needs?

[asyd]

Hi @gulducat thanks your feedback!

I'm indeed able to write a script, it was just a proposal!

[gulducat]

Apologies for the delayed response! I had a nice bit of vacation time recently :)

This turns out to be pretty interestingly complicated. The CLI is doing some acrobatics to accomplish the output per-node, because the data is spread across multiple bits of state. On a little dev cluster with a single allocation, nomad node status -verbose {node-id} produces the following API calls:

/v1/nodes?prefix=6dac5634-b89a-2bb1-542a-1965bf976295 
/v1/node/6dac5634-b89a-2bb1-542a-1965bf976295 
/v1/node/6dac5634-b89a-2bb1-542a-1965bf976295/allocations 
/v1/client/stats?node_id=6dac5634-b89a-2bb1-542a-1965bf976295 
/v1/client/allocation/c4f378ba-b9b4-5934-8411-8a1494e9b4b5/stats 
/v1/agent/self

This logic could potentially be done server-side, but it would be a decent bit of work. And it would still need to go hit each node to request their stats, because that's not stored in server state; on very large clusters, that extra work might be significant.

All that to say, I don't anticipate us tackling this soon, but I will leave it open for future consideration. Other issues, such as #25864 (which also refers to #20252) express a desire to have a more robust way to query cluster state, particularly node information, so maybe one day this issue could be part of such an effort.

We appreciate the proposal. Please do keep them coming!

<3