[Discussion] How should APIs make use of `default_value`?

[rly]

My current interpretation of default_value in the schema language is that, when reading data, if there is no value provided for a particular optional field, then the API or person reading the data should set the field to the default_value.

However, during the HDMF build process, if the user (data writer) does not provide a value for a particular field, HDMF sets the field value to the default_value. As a result, all fields with a default_value will always be written with a value. On read, if a field with a default_value has no value, I believe HDMF does not set the value to the default_value.

The HDMF class generator and the custom API classes for PyNWB also commonly set the default value for arguments in the init docval to the default value in the schema.

I believe MatNWB sets the field value to the default_value on initialization of a type.

As a result, a data reuser cannot tell the difference between the field value being explicitly set to the default value vs it not being explicitly set. I think there is value in differentiating between the two. If it was automatically set by the schema and API, as a data reuser, then I do not 100% trust that the value is correct - the data writer may not have understood the documentation fully. I think this can happen easily and has happened for data with fixed values, e.g., NWB electrical series data not being stored in volts.

LinkML's schema language supports "ifabsent" which I believe follows my interpretation of default_value. This is different from how Pydantic handles default values, but that

There are a handful of places where default_value is used in the NWB schema (none in the HDMF Common Schema): https://github.com/search?q=repo%3ANeurodataWithoutBorders%2Fnwb-schema%20default_value&type=code . NWB extensions may also use default_value. There are a couple places where a TimeSeries.unit is required and has a default value, which is inconsistent with my above interpretation.

If we change the APIs to not write the default_value and to use the default_value on read, then the API user still does not know whether no value was explicitly set in the data unless they manually inspect the data. So changing the API does not fully help the situation. But at least there is a way to differentiate the explicit and implicit cases.

If we eventually want to use LinkML as a schema language, then we will need to resolve this inconsistency between how the APIs currently handle default_value and how LinkML defines default values using ifabsent.

[rly]

The team discussed this issue a few weeks ago. Our conclusion is different from what I wrote above. default_value in the schema informs the user and the APIs what the most common value is in practice. APIs may use this to provide a reasonable default value for users creating data, and in the API, users may override the default value with another value. default_value is not the same as LinkML's ifabsent - it does not mean that this value should be used if no value is present. The schema language does not have an equivalent concept.

The term "default value" is unclear. Perhaps a better term would be "common_value".

default_value does not interact with the required or quantity field - both required and optional attributes and datasets may have a default value.

Note that this still means that a data reuser does not know whether the data creator set this value explicitly or it was set by default in the API. But no matter what we do, there is no good way of gleaning a user's intentions without requiring them to explicitly specify every field, even ones that are a particular value for the vast majority of use cases. This tediousness is not worth it.

In PyNWB currently, the default value in the schema is used as the default value when constructing an object. However, currently it cannot be set to None: NeurodataWithoutBorders/pynwb#2105 . This behavior should be changed so that a user can prevent the field from being written.

In addition, as I wrote in the OP above, if a field in the schema has a default_value and the user supplies a None value, the HDMF object mapper changes the value from None to the default value. This behavior should also be changed so that a field can be set to None to prevent the field from being written.