No_tag columns in the exposure

[micheles]

As requested by @CatalinaYepes . See also #7885.

[raoanirudh]

The engine implicitly filling in missing tag values in a user's exposure could lead to unexpected secondary issues, so I'm not fully in favor of this change. Writing below for future reference the different cases that could arise with missing columns or missing values for certain columns in the exposure:

• A column is included in in the exposure.xml, but the column is absent in one or more of the exposure.csv files. This case should result in an error, the current behavior is ok:
  https://github.com/gem/oq-engine/blob/engine-3.23/openquake/risklib/asset.py#L1104-L1110
• A column is included in in the exposure.xml, and the column is present in all of the exposure.csv files, but some values for this tag are missing in one or more of the exposure.csv files. Here we have two subcases:
  • The tag with some missing values is included in aggregate_by: This case should result in an error. This is not fully handled currently, see: https://github.com/gem/oq-engine/blob/engine-3.23/openquake/risklib/asset.py#L389-L395
  • The tag with some missing values is not included in aggregate_by, or the calculation does not have aggregate_by: This case should run without the engine complaining.

[micheles]

but some values for this tag are missing in one or more of the exposure.csv files

By this you mean that the tag value is the string "No_tag"? Should the engine check for that and raise an error?

[raoanirudh]

but some values for this tag are missing in one or more of the exposure.csv files

By this you mean that the tag value is the string "No_tag"? Should the engine check for that and raise an error?

Not the string "No_tag", but anything that would be flagged by isna().

[micheles]

@raoanirudh perhaps you are missing the fact that when reading the CSV files the engine uses keep_default_na=False which means that there are NEVER NaNs in the exposure. The origin of the NaNs can be only one, pandas.concat() when concatenating the dataframes associated to the CSV files, which adds NaNs for missing columns and such columns can then be filled with No_tag automatically. There is no risk of confusing the case of a missing column with the case of a column being present with some NaN value, because if a column is present all of its values are not NaN (at the level of a single CSV file).