CZO WOF 1.x and SiteType conundrum

[emiliom]

We recently realized (at Nov 6 CZOData call?) that the CZOCentral-hosted WOF services and backend ODM databases were all based on 1.0 versions. As @horsburgh pointed out (and documented here), a shortcoming of 1.0 is that it doesn't support a SiteType CV assignment. @tom2275 recently completed the upgrade of all these services to 1.1 (Thanks!!).

But it was also pointed out that the original CZO Shared Vocabulary never supported a SiteType CV. So, unsurprisingly, the upgraded WOF 1.1 services don't seem to have SiteType content; here's a sample GetSiteInfo REST response (BTW, the GetSiteInfo response doesn't include a SiteType element at all; that seems like an oversight). So, we now have the capacity to store and convey SiteType, but it's unlikely that any of the CZO services and sites have been assigned one.

Downstream usage is impacted, specifically in the CZO vizer application. We can't assign site type and site type map icons automatically based on the CZO services. The only possible options I see are developing a set of "empirical" rules based on variables available and associated generalCategory and sampleMedium, but this could get complicated; or working with CZO data managers and having them assign manually, on a spreadsheet of sites using site codes as used in the WOF services, a site type based on a CV provided by us (from ODM2 SiteTypeCV, hopefully?).

Thoughts, any one? @aufdenkampe, @izaslavsky? I'd include David Lubinski, but don't know if he has a github account.

[horsburgh]

@emiliom - SiteType is optional in ODM 1.1.1. I believe that the convention of WaterML 1.1 is that if an element doesn't exist (e.g., a SiteType is not specified in the ODM database), that element is not included in the WaterML response. Ilya and Tom can clarify if this is inaccurate. If SiteType is specified in the ODM database it will be conveyed in the WaterML response - although it is specified as a siteProperty element. For example, see:

http://data.iutahepscor.org/loganriverwof/REST/waterml_1_1.svc/datavalues?location=iutah:LR_WaterLab_AA&variable=iutah:WaterTemp_EXO&startDate=2014-11-18T03:45:00Z&endDate=2014-11-21T03:45:00Z

To get it right, we probably need to work with the CZO Data managers to assign site types.

[emiliom]

Thank you, @horsburgh! Very useful. I didn't know about those WaterML 1.1. conventions ("if an element doesn't exist (e.g., a SiteType is not specified in the ODM database), that element is not included in the WaterML response"). @izaslavsky and @tom2275, please confirm. Regardless, it looks like it's not being applied consistently in the wild:

• In this czo_merced GetSiteInfo REST response from CZO Central, <siteProperty name="County"> and <siteProperty name="State"> are included but are empty, whereas an empty <siteProperty name="Site Type"> is not included.
• In Jeff's example (a GetValues REST response), <siteProperty name="Site Type"> is included and populated (same for County and State), whereas <siteProperty name="Site Comments"> is included but empty.

Oh well. Not a huge deal, and can be mitigated.

More importantly:

To get it right, we probably need to work with the CZO Data managers to assign site types.

Thanks. I didn't bring up that obvious 3rd option. That's the long-term, necessary solution, of course. I'm just not sure how quickly it can be implemented if it involves asking all data managers to update all their display files (or just their header files?), and ensuring Tom's display-file-to-ODM1.1 ingester can process updates to that kind of metadata. Plus, if we're telling data managers that we're working on "display file version 2" and ODM2, is it reasonable to ask them to make this change in the next couple of months, when it may all change again in a few months?

No easy solutions.

[izaslavsky]

if Emilio needs sitetype for the interface, we may as well make it mandatory in the "CZO profile" of ODM. How many sites are we talking about? If the CV for sitetypes is not too complex then it can be done fairly quickly i hope - directly in the database? It may be also useful to do this in view of future update to ODM2
Sent using CloudMagic

[emiliom]

Thanks, Ilya. To clarify, we can certainly do the visualization portal w/o site type (ie, all sites from CZO WOF end points would be classified the same way). But it would be a lot less valuable and user friendly.

As for how many sites we're talking about, Tom can get a definitive answer, but my current guesstimate is about 500-800, spread out across ~7 ODM databases.

Your suggestion to "do it in the database" raises an interesting, pragmatic, semi-short-term middle way. Here are the steps:

1. We come up with a reasonably simple set of site types, inspired by our current ODM2 CV discussions.
2. We give each CZO data manager a site list spreadsheet that includes the site codes used in their ODM database / WOF services. They do the manual assignment, based on site types from CZO WOF 1.x and SiteType conundrum #1.
3. Tom uses those spreadsheets to update the ODM databases (after validating and potentially manually correcting site type entries).

Once 3 is done and SiteType is available in WOF 1.1 GetSiteInfo responses, I'm good to go.

I'll still move forward initially assuming no site type differentiation for CZOCentral WOF sites; we could call it something like "CZO Time Series Site".

[izaslavsky]

Emilio, what you suggest makes a lot of sense to me. I expect that
(assuming the CV is fairly small), most sites would fall into just a
handful of rubrics. The steps are exactly right. We'd also need updated
instructions for data managers regarding the sitetypes - and at some point
announce that Tom will now validate display files against the sitetype CV
as well.

On Fri, Nov 21, 2014 at 1:06 PM, Emilio Mayorga notifications@github.com
wrote:

Thanks, Ilya. To clarify, we can certainly do the visualization portal w/o
site type (ie, all sites from CZO WOF end points would be classified the
same way). But it would be a lot less valuable and user friendly.

As for how many sites we're talking about, Tom can get a definitive
answer, but my current guesstimate is about 500-800, spread out across ~7
ODM databases.

Your suggestion to "do it in the database" raises an interesting,
pragmatic, semi-short-term middle way. Here are the steps:

1. We come up with a reasonably simple set of site types, inspired by
   our current ODM2 CV discussions.
2. We give each CZO data manager a site list spreadsheet that includes
   the site codes used in their ODM database / WOF services. They do the
   manual assignment, based on site types from CZO WOF 1.x and SiteType conundrum #1
   CZO WOF 1.x and SiteType conundrum #1.
3. Tom uses those spreadsheets to update the ODM databases (after
   validating and potentially manually correcting site type entries).

Once 3 is done and SiteType is available in WOF 1.1 GetSiteInfo responses,
I'm good to go.

I'll still move forward initially assuming no site type differentiation
for CZOCentral WOF sites; we could call it something like "CZO Time Series
Site".

—
Reply to this email directly or view it on GitHub
#1 (comment)
.

[emiliom]

Thanks, Ilya. You're right that, in parallel, we'll want to set up (and announce) the process to include and validate new display files against a sitetype CV.