API to manage user controlled metadata as extended attribute.

[jcdubois]

Hello guys,

For our purpose we were in need of a way to add user controlled metadata to files stored inside the reliance edge filesystem (posix version).

The need was for these data to be associated to the file but distinct from the file data payload. Moreover it was desired that these metadata could be protected in integrity.

So instead of devising a solution strictly on top of the actual reliance API, I started to add a very crude/simple "user controlled metadata API" loosely inspired by the linux/posix "extended attribute API".

Here, the feature is rather limited as:

• the max number of attributes is determined at compile time,
• the size of an attribute is fixed at 32 bits
• the attributes are referenced by index rather than by a key.

The drawback is that the API is not really linux compliant. Anyway there is no posix version of this API it seems. So maybe it doesn't really matter to have a Reliance specific API.

My present prototype is available on a branch in my github repository (https://github.com/jcdubois/reliance-edge/tree/attr).

The feature is optional (if REDCONF_ATTRIBUTES_MAX is equal to 0 it is not included) and it allows the user to store 32 bits user controlled values inside dir/file inodes. Because it is stored in the inode, it is protected by the inode CRC.

I would like to know if such feature is of interest to you and if you would consider merging it (after careful review and required fix/change) to reliance main line.

Thanks.

JC

Note: on my branch I have added a crc_file_wrapper API as an example of what could be achieved with this feature (using one attribute to store the data file CRC). This API is not necessarily candidate for inclusion (unless you find it useful). Extended attributes can be used to store a lot of other things related to file like owner ID, file type, encoding, identifier, ... and any number of other tags.

Note: I ported fsstress to use the crc_file_wrapper API (rather than the red API) to test it some bit and it did work as expected (with some performance hit when writing files).

[danielrlewis]

I would like to know if such feature is of interest to you and if you would consider merging it (after careful review and required fix/change) to reliance main line.

Our team will add this to our feature list for consideration. We will need to decide whether this is useful enough to warrant changing the on-disk layout. Let me explain what I mean by the "changing the on-disk layout"...

Reliance Edge records all the configuration settings which affect its on-disk metadata in the master block: see the MASTERBLOCK structure in core/include/rednodes.h. This allows Reliance Edge to validate that its compile-time configuration is correct for the volume: see RedVolMountMaster() in core/driver/volume.c. Theoretically, this also allows for tools (or even alternate implementations of the driver) which can work with any Reliance Edge volume, regardless of redconf.[ch] settings, by reading the master block to discover the configuration.

Your patch adds a new configuration option which affects the interpretation of the on-disk metadata (specifically, the inode). That option, the REDCONF_ATTRIBUTES_MAX value, would need to be stored in the master block, for the reasons described above. We would also need to prevent older versions of the file system, which would not know about the feature, from mounting a volume which has it enabled; merely appending a new field onto the master block would not accomplish that, so we would also need to increment the on-disk layout number (or set one of the currently-unused master block flag values). Such master block changes would be required for any feature like this -- but, to be clear, I am not asking for any work on your part at this point, since we have not committed to merging the feature.

As an aside, partly for anyone else reading this, I will note that if we do update the on-disk layout, the driver will retain support for the older on-disk layout.

[jcdubois]

Hello Daniel,
thanks for the explanation on the "changing the on-disk layout" rule/requirement.
I was not aware of it but it does make sense.
Let me know if you need anything else from me.
JC