Allow specifying column metadata for features in FeatureGenerator

[kosack]

Please describe the use case that requires this feature.

FeatureGenerator transforms an input astropy Table to an output, generating new columns. In some cases, the resulting table will be written out to a FITS for ECSV file. The column description is carried over to the output table for original columns, but there is no way to specify it for generated columns.

Describe the solution you'd like

An (optional) way to give a column description to generated feature columns.

Simplest implementation with no change to current API: allow a third element in the Tuple used to specify a feature, i.e.:

FeatureGenerator:
  features:
     - ["column_with_description", "x**2+3*x", "the description goes here"]
     - ["some_other_col", "x*2"]   # the old API should still work, assigning a default blank description

Describe alternatives you've considered

An alternative would be a more radical change, using a list[dict] in a defined format. It's more readable, but would be a large change and break the API, so I think it's better to stick with the simple option.

FeatureGenerator:
  features:
    - name: "column_with_description"
      expression: "x**2+3*x"
      description: "the description goes here"
    - name: some_other_col
      expression: "x**2"

Additional context

This is something I would like to use to greatly simplify DL2EventLoader, as in #2919, which currently has a very complex way of specifying metadata for new columns.

[maxnoe]

I would prefer the more expressive dict-base syntax, it's also easier to extend later with additional fields, might we need them.

I think at least "unit" should be added already now.

[maxnoe]

Edit: since inputs to the expressions have units, the output also will have one. So unit should not be needed.

[kosack]

Edit: since inputs to the expressions have units, the output also will have one. So unit should not be needed.

Yes, units get propagated already (well at least with #2921). I think then the only other column metadata we would want to store is the description, which I think is useful for understanding the output. However, perhaps there is a case for other metadata? I do prefer the more structured version, but that is also a large change to the API: all current config files would need updating. Granted, i don't really like how YAML serializes list[tuple] so it may be worth changing to list[dict]:

Current config using list[tuple] is quite ugly:

print(yaml.safe_dump(feature_generator.get_current_config()))

FeatureGenerator:
  features:
  - - gh_score
    - RandomForestClassifier_prediction
  - - true_fov_offset
    - angular_separation(reco_fov_lon, reco_fov_lat, 0*u.deg, 0*u.deg)
  - - reco_fov_offset
    - angular_separation(true_fov_lon, reco_fov_lat, 0*u.deg, 0*u.deg)
  - - multiplicity
    - np.count_nonzero(RandomForestClassifier_telescopes,axis=1)
...