Report disk size of path.data

[stefnestor]

Description

👋 howdy, team!

Will you kindly consider expanding Node Stats (and potentially CAT Allocation) to include stats about the disk used by Elasticsearch's path.data folder?

This would more directly enable users to investigate disk indices vs cache issues per stat derivation graphic of current vs with this data addition enabled:

This would help with investigations for situations like #104667 .

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[stefnestor]

Noting from investigation, it appears more recent versions do separately report path data stats from total but so far they have reported the same.

data

---

Examples

1. ESS example (random)
   

   01654458_esDiag_2024Jul13T0953$ cat nodes_stats.json | jq -rc '.nodes[]|.name as $n|.fs.total as $t|.fs.data[] as $d|{n:$n, ta:$t.available, tt:$t.total, da:$d.available, dt:$d.total}'
   {"n":"instance-0000000025","ta":"189.6gb","tt":"240gb","da":"189.6gb","dt":"240gb"}
   {"n":"instance-0000000021","ta":"29.2gb","tt":"296gb","da":"29.2gb","dt":"296gb"}
   {"n":"instance-0000000024","ta":"150.8gb","tt":"240gb","da":"150.8gb","dt":"240gb"}
   {"n":"instance-0000000019","ta":"11.9gb","tt":"12gb","da":"11.9gb","dt":"12gb"}
   {"n":"tiebreaker-0000000027","ta":"11.9gb","tt":"12gb","da":"11.9gb","dt":"12gb"}

2. ESS example (overrode disk size)

   tres$ echo "22260d1cc60f47b483ba4176620f3135" > .d
   tres$ ess '_nodes/stats?human' | jq . > nodes_stats.json
   tres$ cat nodes_stats.json | jq -rc '.nodes[]|.name as $n|.fs.total as $t|.fs.data[] as $d|{n:$n, ta:$t.available, tt:$t.total, da:$d.available, dt:$d.total}|select(.tt!=.dt)'
   # NULL 

3. On-prem example (random)

   # ~/01588193/scrubbed-api-diagnostics-20240325-053509-20240401-041630
   $ cat nodes_stats.json | jq -rc '.nodes[]|.name as $n|.fs.total as $t|.fs.data[] as $d|{n:$n, ta:$t.available, tt:$t.total, da:$d.available, dt:$d.total}|select(.tt!=.dt)'
   # NULL

4. On-prem example (random)

   # ~/01611156/local-diagnostics-20240502-083745 
   $ cat nodes_stats.json | jq -rc '.nodes[]|.name as $n|.fs.total as $t|.fs.data[] as $d|{n:$n, ta:$t.available, tt:$t.total, da:$d.available, dt:$d.total}|select(.tt!=.dt)'
   # NULL

[DaveCTurner]

I don't think there's a way to compute this stat unfortunately. The size-on-disk is only exposed by the OS at the filesystem level, with good reason: size-on-disk must take account of all sorts of other factors (e.g. deduplication of hard-linked files) that can't be resolved for proper subsets of the files in a filesystem. I mean I'm sure there's some way to compute this stat, but it'd be very tricky to get right, take an awfully long time to compute on each request, and might require permissions that ES doesn't normally have.

[stefnestor]

That's sad but makes sense.

So to confirm, we believe Elasticsearch can't run commands like ... on current permissions across supported OS's for its used directory?

ls -l /path/to/directory   # for example, ES data path
df -h /path/to/directory   # for example, /mnt/data folder
du /path/to/directory

[DaveCTurner]

ls -l /es/data/path counts file sizes, not disk space. In GET _nodes/stats it's pretty much .nodes.*.indices.store.size_in_bytes plus .nodes.*.indices.translog.size_in_bytes.

df -h /es/data/path gives information about the overall filesystem containing the given path (plus quotas etc), not the size of that subfolder. In GET _nodes/stats it's pretty much .nodes.*.fs.data.*.free_in_bytes.

du /es/data/path does return info about actual disk space but is rather slow and complex, and I'm not sure we have all the permissions needed to do what it does.