Add note about ctx.files memory impact

[hauserx]

Description

Usage of ctx.files in rules can be memory heavy in case of a lot of inputs. According to current documentation using ctx.files is shortcut for

[f for t in ctx.attr.<ATTR> for f in t.files]

and indeed when creating some syntetic benchmarks noticed high memory usage that could be explained by file depset being eagerly resolved.

I have updated documentation with info about it and small sample of creating depset from ctx.attr as an alternative.

I have also found in Kleaf documentation note that also recommends using depsets in all cases but when small number of inputs is guaranteed: https://android.googlesource.com/kernel/build/+/refs/heads/main/kleaf/docs/kleaf_development.md#prefer-depset-over-in-rules

Motivation

When investigating high memory usage for some of our heavy builds (~1M actions) noticed high memory usage in loading/analysis phase on pretty basic synthetic benchmarks. Benchmark is basically 100k-1M small actions, each using the same 1k-10k-100k files. Manipulating number of actions and number of input files shows that around 8 bytes is needed per each occurrence of an input file. So e.g. for 100k actions each using same 10k input files, bazel reports used-heap-size-after-gc: around 8GB after the analysis phase (bazel build --nobuild //...) . Taking heap dumps showed that ~95% of memory is consumed by NestedSets. Using solution described in this PR reduced memory to ~400MB.

Build API Changes

No

Release Notes

RELNOTES: None

[fmeum]

The ideal solution would be to migrate these to return special lists that only flatten the depset when needed. I don't know whether that's feasible though.