Store/retrieve device calibration data

[davidpablocohn]

We need to be able to store/retrieve/use device calibration data:

• calibration value
• date of calibration
• name of tester and notes
• possibly an equation for how that calibration should be used to adjust sensor output values

Transforms should have access to these metadata so that they can perform the adjustment (and pass the metadata of how the adjustment was made along with the new, adjusted values).

There should be mechanisms for associating those data with data streams, such as outputting as header in a log file or including in JSON or DASRecord messages.

[webbpinner]

For what it's worth I do not think this should be handled within a data acquisition system and instead should be handled by a dedicated metadata tool. I see where it would be a slick feature for sensors that are handled by OpenRVDAS, however not all data systems on a research vessel are run through OpenRVDAS.

My personal preference would be to develop a stand-alone sensor metadata tracking system that could be used to manage metadata for all sensor systems on a vessel. I've actually been thinking recently that this metadata tool might be the next project OpenOceanTools takes on.

[davidpablocohn]

It sounds reasonable to me to have an external system manage metadata. But I think OpenRVDAS components need to be able to retrieve/parse/work with it. Would be interesting to consider whether simply having a "MetadataReader" component that could retrieve things like calibration values (to be used, e.g., in a DerivedDataTransform) would be sufficient.

…

On Mon, Oct 21, 2019 at 11:36 AM Webb Pinner ***@***.***> wrote: For what it's worth I do not think this should be handled within a data acquisition system and instead should be handled by a dedicated metadata tool. I see where it would be a slick feature for sensors that are handled by OpenRVDAS, however not all data systems on a research vessel are run through OpenRVDAS. My personal preference would be to develop a stand-alone sensor metadata tracking system that could be used to manage metadata for all sensor systems on a vessel. I've actually been thinking recently that this metadata tool might be the next project OpenOceanTools takes on. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#169?email_source=notifications&email_token=AFO7V3SKJOCYPNV3GHNQVWTQPYAE5A5CNFSM4JC7BXBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB3Q3TI#issuecomment-544673229>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFO7V3SHYAYBYQACLC533VLQPYAE5ANCNFSM4JC7BXBA> .

[webbpinner]

Been thinking about this a bit more. I still believe a DAS should just deal with reading and writing data. Transforms for things like prepending timestamps and true-wind calculations are a necessary complication but I believe anything beyond that should be handled by an external mechanism.

All that said, I am now in favor of having a transform that can apply a user-defined lambda function to a data stream. This would allow an operator to perform static corrections (i.e. adding the distance from a depth sensor to the water line for proper depth-from-surface) and unit conversions (knots <--> kph, C <--> F, etc). This type of transform could also be used to apply a calibration curve if the operator so desired. I can't take credit for this idea. An older version of the DAS used on WHOI ships included this functionality. I think SCS can do this as well.

My apprehension about getting into automated corrections based on calibration data comes from several bad experiences and was recently reinforced by a incident where a piece of mid-water equipment made contact with the seafloor because the wrong correction equation was being unknowingly applied in real-time to a depth sensor.

With regards to metadata tracking... I hope you can talk with Chris Romsos about work he's done in this area that I just uncovered yesterday.

[davidpablocohn]

Yup - this kind of conversion is what the DerivedDataTransform is all about. Of course, OpenRVDAS should (always) be storing the raw values that come out of the sensors; I just want to make sure it has access to the various calibration factors that let it turn those raw values into numbers that are meaningful to the scientists and crew on board.

…

On Tue, Oct 22, 2019 at 8:38 AM Webb Pinner ***@***.***> wrote: Been thinking about this a bit more. I still believe a DAS should just deal with reading and writing data. Transforms for things like prepending timestamps and true-wind calculations are a necessary complication but I believe anything beyond that should be handled by an external mechanism. All that said, I am now in favor of having a transform that can apply a user-defined lambda function to a data stream. This would allow an operator to perform static corrections (i.e. adding the distance from a depth sensor to the water line for proper depth-from-surface) and unit conversions (knots <--> kph, C <--> F, etc). This type of transform could also be used to apply a calibration curve if the operator so desired. I can't take credit for this idea. An older version of the DAS used on WHOI ships included this functionality. I think SCS can do this as well. My apprehension about getting into automated corrections based on calibration data comes from several bad experiences and was recently reinforced by a incident where a piece of mid-water equipment made contact with the seafloor because the wrong correction equation was being unknowingly applied in real-time to a depth sensor. With regards to metadata tracking... I hope you can talk with Chris Romsos about work he's done in this area that I just uncovered yesterday. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#169?email_source=notifications&email_token=AFO7V3QVEDQLHVNF3O5EBILQP4UBJA5CNFSM4JC7BXBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6MX2I#issuecomment-545049577>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFO7V3WDT2CHLSUGFPMNFZDQP4UBJANCNFSM4JC7BXBA> .

[LMG-ET]

Yes.... I see Webb's point, but I consider it apples and oranges. Both tasty fruit, but different. Let's take a common example: A transmissometer. You've got calibration factors you copy from the vendor's calibration sheet. And WAY YES! It would be cool to have a tool where I could upload that and it would get indexed with make/model/serial-number/date, when I installed it, etc. Especially if it had a API I could query. But to get the actual beam transissance (is that the right word?) I need those factors, plus ideally in-situ dark and air measurements (which are data points sent to the DAS via the sea-water wall data stream so that the values used are in the raw data somewhere). The meta-data system wouldn't necessarily know about those since they'll change every time we clean the instrument and sample the air/dark values, and let's face it, tech's get busy and don't always have time to futz with metadata.

So yes... a function using (basically) the derived data transform is great. I think it might be hard to figure out how to enter that into the existing config structure though. I'd almost suggest a separate config file and a special-case thing like the true-winds transform that reads it. That way you could describe the metadata, the equation(s), the required data from the DAS, and the data you're providing TO the das. Since such a transform might apply to more than one logger stream, you maybe can't just it in the chain for one instrument.

I've actually been working on something like that, and can't wait for Pablo to show up so I can have him tell me I'm doing it wrong and show me the OpenRVDAS way. :-)

[davidpablocohn]

Sounds good - looking forward to seeing what you've got in mind.

…

On Tue, Jul 20, 2021, 4:26 PM LMG-ET ***@***.***> wrote: Yes.... I see Webb's point, but I consider it apples and oranges. Both tasty fruit, but different. Let's take a common example: A transmissometer. You've got calibration factors you copy from the vendor's calibration sheet. And WAY YES! It would be cool to have a tool where I could upload that and it would get indexed with make/model/serial-number/date, when I installed it, etc. Especially if it had a API I could query. But to get the actual beam transissance (is that the right word?) I need those factors, plus ideally in-situ dark and air measurements (which are data points sent to the DAS via the sea-water wall data stream so that the values used are in the raw data somewhere). The meta-data system wouldn't necessarily know about those since they'll change every time we clean the instrument and sample the air/dark values, and let's face it, tech's get busy and don't always have time to futz with metadata. So yes... a function using (basically) the derived data transform is great. I think it might be hard to figure out how to enter that into the existing config structure though. I'd almost suggest a separate config file and a special-case thing like the true-winds transform that reads it. That way you could describe the metadata, the equation(s), the required data from the DAS, and the data you're providing TO the das. Since such a transform might apply to more than one logger stream, you maybe can't just it in the chain for one instrument. I've actually been working on something like that, and can't wait for Pablo to show up so I can have him tell me I'm doing it wrong and show me the OpenRVDAS way. :-) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#169 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFO7V3XH2EYU25B4XXFUROLTYXLZDANCNFSM4JC7BXBA> .