normalize fields for OT protocols supported by Arkime

[mmguero]

Arkime has several more OT protocols now. We should go through and add field normalizations in our Logstash pipeline so we can cross-filter between the Arkime field names and those in dashboards.

[sercanokur]

Hi, I’d be happy to take this on.

Could you please assign this issue to me? My plan is to review the current Arkime-to-dashboard field mappings in the Logstash pipeline, identify the missing OT protocol field normalizations, and submit a focused PR.

Thanks!

[mmguero]

Thanks for offering! Here's where I'd look for things:

• Arkime's db.pl's sessions3Update should give you the field mappings you're looking for, or just look at the field list of the Arkime help page
• Unfortunately in order for our Dashboards stuff to know about these field mappings, we need to add them to our composable template. What that means is that anything you identify in db.pl that we're going to normalize to should be duplicated here in dashboards/templates/composable/component/arkime.json
• For the normalization itself, probably the best place to put the mutations is in logstash/pipelines/zeek/1200_zeek_mutate.conf. There are sections here for each (or at least, most) of the Zeek logs (which is pretty much where all the rest of our OT parsing is coming from). If you search for arkime in that file you will see other places where I'm doing this sort of thing.
• When it comes to the dashboards themselves, each dashboard has an ID associated with it in dashboards/dashboards; so, for example, BACnet is dashboards/dashboards/2bec1490-eb94-11e9-a384-0fcf32210194.json. If we want to do the normalization "all the way" what we probably want to do is find references to the field names we normalized to in the dashboard and replace those references with the Arkime field name (so that when we pivot from Dashboards to Arkime it "just works"). Don't worry about changing the Zeek field names that don't have equivalent new fields in Arkime.

Hopefully that can get you started.