Allow (parameterized) reusable logger definitions in cruise definition file

[davidpablocohn]

Much the way devices and device types are reusable, the overhead of creating and maintaining a cruise definition file could be simplified if it was possible to define a few parameterized logger types and reuse them.

It would operate along the lines of jinja, running a preprocessor over the initial file (perhaps automatically when the templatized file is loaded?) to expand it. We discussed some syntax conventions that seem like they would make this scale better than raw Jinja.

[webbpinner]

My thoughts...
Combining config definitions into the loggers section makes sense.

Field name patterns for devices makes sense... I personally would prefer the default pattern to be %device%_%field%

Still on the fence with templates... I find the yaml native variable expansion to be a real improvement over what I've done in the past. That said... I guess one way to implement would be:

Here's what goes in logger_config.yaml:

---
loggers:
  cwnc:
    configs:
    - cwnc-net:
      template: serial_to_net
        kwargs:
          prefix: %prefix%
          baudrate: %baudrate%
          serial_port: %serial_port%

Here's the template... written in it's own file(s) that are included with the logger_config via 'includes' OR written inline within the logger_config.yaml within a templates section:

---
templates:
  serial_to_net:
    readers:
    - class: SerialReader
      kwargs:
        baudrate: %baudrate%
        port: %serial_port%
    transforms:
    - class: TimestampTransform
    - class: PrefixTransform
      kwargs:
        prefix: %prefix%
    writers:
    - class: UDPWriter
      kwargs:
        port: 6224
        destination: 255.255.255.255

So the complete logger_config.yaml would look something like:

---
########################################
loggers:
  PCOD:
    configs:
    - PCOD-off: {}
    - PCOD-net:
      template: serial_to_net
        kwargs:
          prefix: PCOD
          baudrate: 4800
          serial_port: /tmp/tty_PCOD
  cwnc:
    configs:
    - cwnc-off: {}
    - cwnc-net:
      template: serial_to_net
        kwargs:
          prefix: cwnc
          baudrate: 19200
          serial_port: /tmp/tty_cwnc

########################################
templates:
  serial_to_net:
    readers:
    - class: SerialReader
      kwargs:
        baudrate: %baudrate%
        port: %serial_port%
    transforms:
    - class: TimestampTransform
    - class: PrefixTransform
      kwargs:
        prefix: %prefix%
    writers:
    - class: UDPWriter
      kwargs:
        port: 6224
        destination: 255.255.255.255

[davidpablocohn]

Thanks! Yeah, that's pretty close to what Ella proposed - I've got a couple of (my version of) her sample files I was going to add to the feature request as soon as I cleaned up my typos.

…

On Sun, Feb 23, 2025 at 8:54 PM Webb Pinner ***@***.***> wrote: My thoughts... Combining config definitions into the loggers section makes sense. Field name patterns for devices makes sense... I personally would prefer the default pattern to be %device%_%field% Still on the fence with templates... I find the yaml native variable expansion to be a real improvement over what I've done in the past. That said... I guess one way to implement would be: Here's what goes in logger_config.yaml: --- configs: cwnc-net: template: serial_to_net kwargs: prefix: %prefix% baudrate: %baudrate% serial_port: %serial_port% Here's the template... written in it's own file(s) that are included with the logger_config via 'includes' OR written inline within the logger_config.yaml within a templates section: --- templates: serial_to_net: readers: - class: SerialReader kwargs: baudrate: %baudrate% port: %serial_port% transforms: - class: TimestampTransform - class: PrefixTransform kwargs: prefix: %prefix% writers: - class: UDPWriter kwargs: port: 6224 destination: 255.255.255.255 So the complete logger_config.yaml would look something like: --- ######################################## loggers: PCOD: configs: - PCOD-off: {} - PCOD-net: template: serial_to_net kwargs: prefix: PCOD baudrate: 4800 serial_port: /tmp/tty_PCOD cwnc: configs: - cwnc-off: {} - cwnc-net: template: serial_to_net kwargs: prefix: cwnc baudrate: 19200 serial_port: /tmp/tty_cwnc ######################################## templates: serial_to_net: readers: - class: SerialReader kwargs: baudrate: %baudrate% port: %serial_port% transforms: - class: TimestampTransform - class: PrefixTransform kwargs: prefix: %prefix% writers: - class: UDPWriter kwargs: port: 6224 destination: 255.255.255.255 — Reply to this email directly, view it on GitHub <#422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFO7V3VMKYPGL5AG2K3ZOOL2RGLGNAVCNFSM6AAAAABXV5WNAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNZWG42DONJYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: webbpinner]*webbpinner* left a comment (OceanDataTools/openrvdas#422) <#422 (comment)> My thoughts... Combining config definitions into the loggers section makes sense. Field name patterns for devices makes sense... I personally would prefer the default pattern to be %device%_%field% Still on the fence with templates... I find the yaml native variable expansion to be a real improvement over what I've done in the past. That said... I guess one way to implement would be: Here's what goes in logger_config.yaml: --- configs: cwnc-net: template: serial_to_net kwargs: prefix: %prefix% baudrate: %baudrate% serial_port: %serial_port% Here's the template... written in it's own file(s) that are included with the logger_config via 'includes' OR written inline within the logger_config.yaml within a templates section: --- templates: serial_to_net: readers: - class: SerialReader kwargs: baudrate: %baudrate% port: %serial_port% transforms: - class: TimestampTransform - class: PrefixTransform kwargs: prefix: %prefix% writers: - class: UDPWriter kwargs: port: 6224 destination: 255.255.255.255 So the complete logger_config.yaml would look something like: --- ######################################## loggers: PCOD: configs: - PCOD-off: {} - PCOD-net: template: serial_to_net kwargs: prefix: PCOD baudrate: 4800 serial_port: /tmp/tty_PCOD cwnc: configs: - cwnc-off: {} - cwnc-net: template: serial_to_net kwargs: prefix: cwnc baudrate: 19200 serial_port: /tmp/tty_cwnc ######################################## templates: serial_to_net: readers: - class: SerialReader kwargs: baudrate: %baudrate% port: %serial_port% transforms: - class: TimestampTransform - class: PrefixTransform kwargs: prefix: %prefix% writers: - class: UDPWriter kwargs: port: 6224 destination: 255.255.255.255 — Reply to this email directly, view it on GitHub <#422 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFO7V3VMKYPGL5AG2K3ZOOL2RGLGNAVCNFSM6AAAAABXV5WNAWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNZWG42DONJYGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[davidpablocohn]

For including:

# Anything here is expected to be a YAML dict that gets merged/appended at the top level
# with this dict. If there are duplicate keys then:
#   - if value is scalar, it gets overwritten
#   - if value is list, it gets appended
#   - if value is dict, it gets merged
includes:
    /opt/openrvdas/investigator/configs/logger_templates/*.yaml

A couple of possibilities for templating, riffing off both Webb and Ella's ideas, depending on how much templatting we'd like to support. In theory, we could support both:

variables:
  source_ip: 224.0.36.0
  eol: '\n'
  udp_loggers:
    - pco2
    - knud
    - s330

modes:
      '%udp_loggers%': '%udp_loggers%-off'
      met: met-off
      pco2: pco2-off
    log+db:
      '%udp_loggers%': '%udp_loggers%-on'
      met: met-on
      pco2: pco2-on

loggers:
  pco2:
    logger_template: standard_udp
    variables:
      port: 54321
  ...

Alternatively, could use variable expansion all the way down:

variables:
  source_ip: 224.0.36.0
  eol: '\n'

  # or,alternatively, each could be a dict with its own defs:
  udp_loggers:
    pco2:
      logger_template: standard_udp
      logger_port: 54321
    knud:
      logger_template: standard_udp
      logger_port: 54322
    s330:
      logger_template: standard_udp
      logger_port: 54323

modes:
      '%udp_loggers%': '%udp_loggers%-off'
      met: met-off
      pco2: pco2-off
    log+db:
      '%udp_loggers%': '%udp_loggers%-on'
      met: met-on
      pco2: pco2-on

loggers:
  # Gets expanded using variables
  '%udp_loggers%':
    logger_template: '%logger_template%'
    port: '%logger_port%'

[webbpinner]

I'd like to see more complete examples. I'm unclear how the examples in that last post will scale for some of the more complicated installations like Falkor, OceanXplorer and Revelle.

I'm also not feeling the modes section syntax. What about including the modes in the logger config as a scalar or array:

  cwnc:
    configs:
    - cwnc-off:
        modes: off
    - cwnc-net:
        modes: [net, net+influx]

the modes section could then be built automatically from the logger configs

[webbpinner]

ok... this is gonna be a long one but I needed to piece together a complete example based on the NBP config. Here's where I got:

Here's the sample logger config:

---
net_writer: &net_writer_defaults
  port: 6224
  destination: 255.255.255.255

cwnc: &cwnc_defaults
  prefix: cwnc
  baudrate: 19200
  serial_port: /tmp/tty_cwnc

pcod: &pcod_defaults
  prefix: pcod
  baudrate: 4800
  serial_port: /tmp/tty_PCOD

########################################
loggers:

  pcod:
    configs:
    - pcod-off: { modes: off }
    - pcod-net:
        modes: [net, net+influx]
        template:
          serial_to_net:
            kwargs:
              <<: *pcod_defaults
    - pcod-net+file:
        modes: [net+file, net+file+influx]
        template:
          serial_to_net+file:
            kwargs:
              <<: *pcod_defaults
              filepath: /var/tmp/log/pcod/raw/<<cruise>>_pcod

  cwnc:
    configs:
    - cwnc-off: { modes: off }
    - cwnc-net:
        modes: [net, net+influx]
        template:
          serial_to_net:
            kwargs:
              <<: *cwnc_defaults
    - cwnc-net+file:
        modes: [net+file, net+file+influx]
        template:
          serial_to_net+file:
            kwargs:
              <<: *cwnc_defaults
              filepath: /var/tmp/log/cwnc/raw/<<cruise>>_cwnc

  influx:
    configs:
    - influx-off: { modes: [ off, net, net+file] }
    - influx-on: { modes: [net+influx, net+file+influx] }
      readers:
        - class: UDPReader
          <<: *net_writer_defaults
      writers:
        - class: InfluxDBWriter


########################################
templates:
  serial_to_net: &serial_to_net
    readers:
    - class: SerialReader
      kwargs:
        baudrate: <<baudrate>>
        port: <<serial_port>>
    transforms:
    - class: TimestampTransform
    - class: PrefixTransform
      kwargs:
        prefix: <<prefix>>
    writers:
    - class: UDPWriter
      kwargs:
        <<: *net_writer_defaults

  serial_to_net+file:
    <<: *serial_to_net
    transforms:
    - class: TimestampTransform
    writers:
    - class: LogfileWriter
      kwargs:
        filebase: <<filebase>>
    - class: ComposedWriter
      kwargs:
        transforms:
        - class: PrefixTransform
          kwargs:
            prefix: <<prefix>>
        writers:
        - class: UDPWriter
          kwargs:
            <<: *net_writer_defaults

This is 100% valid yaml confirmed by https://www.yamllint.com. I opt'ed to replace the %variable_name% syntax with <<variable_name>> as yaml doesn't like values that start with a % character. Taking this into account is actually important as ship's use yaml linting tools to proofread their logger configs all the time.

When you use YAML's default alias resolver it returns as:

---
net_writer:
  port: 6224
  destination: 255.255.255.255
cwnc:
  prefix: cwnc
  baudrate: 19200
  serial_port: /tmp/tty_cwnc
pcod:
  prefix: pcod
  baudrate: 4800
  serial_port: /tmp/tty_PCOD
loggers:
  pcod:
    configs:
      - pcod-off:
          modes: off
      - pcod-net:
          modes:
            - net
            - net+influx
          template:
            serial_to_net:
              kwargs:
                prefix: pcod
                baudrate: 4800
                serial_port: /tmp/tty_PCOD
      - pcod-net+file:
          modes:
            - net+file
            - net+file+influx
          template:
            serial_to_net+file:
              kwargs:
                prefix: pcod
                baudrate: 4800
                serial_port: /tmp/tty_PCOD
                filepath: /var/tmp/log/pcod/raw/<<cruise>>_pcod
  cwnc:
    configs:
      - cwnc-off:
          modes: off
      - cwnc-net:
          modes:
            - net
            - net+influx
          template:
            serial_to_net:
              kwargs:
                prefix: cwnc
                baudrate: 19200
                serial_port: /tmp/tty_cwnc
      - cwnc-net+file:
          modes:
            - net+file
            - net+file+influx
          template:
            serial_to_net+file:
              kwargs:
                prefix: cwnc
                baudrate: 19200
                serial_port: /tmp/tty_cwnc
                filepath: /var/tmp/log/cwnc/raw/<<cruise>>_cwnc
  influx:
    configs:
      - influx-off:
          modes:
            - off
            - net
            - net+file
      - influx-on:
          modes:
            - net+influx
            - net+file+influx
        readers:
          - class: UDPReader
            port: 6224
            destination: 255.255.255.255
        writers:
          - class: InfluxDBWriter
templates:
  serial_to_net:
    readers:
      - class: SerialReader
        kwargs:
          baudrate: <<baudrate>>
          port: <<serial_port>>
    transforms:
      - class: TimestampTransform
      - class: PrefixTransform
        kwargs:
          prefix: <<prefix>>
    writers:
      - class: UDPWriter
        kwargs:
          port: 6224
          destination: 255.255.255.255
  serial_to_net+file:
    readers:
      - class: SerialReader
        kwargs:
          baudrate: <<baudrate>>
          port: <<serial_port>>
    transforms:
      - class: TimestampTransform
    writers:
      - class: LogfileWriter
        kwargs:
          filebase: <<filebase>>
      - class: ComposedWriter
        kwargs:
          transforms:
            - class: PrefixTransform
              kwargs:
                prefix: <<prefix>>
          writers:
            - class: UDPWriter
              kwargs:
                port: 6224
                destination: 255.255.255.255

Notice how with zero code changes and leveraging YAML aliases we've drastically eliminated repeating things like port numbers, serial ports and baud rates.

You also have all the data you need for fleshing out templates as a standard kwargs yaml object.

...and now, through the magic of chatgpt... here's the python code that can take this yaml and build the standard OpenRVDAS modes object:

import yaml
from collections import defaultdict

# YAML content (the string you provided)
yaml_data = """
<<INSERT OUTPUT FROM ABOVE>>
"""

# Load the YAML data
data = yaml.safe_load(yaml_data)

# Create a dictionary to store loggers by modes
modes_dict = defaultdict(dict)

# Iterate over loggers and configs to categorize them by modes
for logger, logger_data in data['loggers'].items():
    for config in logger_data['configs']:
        for config_name, config_data in config.items():
            if 'modes' in config_data:
                modes = config_data['modes']
                # If 'modes' is a list, append the config key to each mode
                if isinstance(modes, list):
                    for mode in modes:
                        modes_dict[mode][logger] = config_name
                else:
                    modes_dict[modes][logger] = config_name

# Wrap the result inside the 'modes' parent object
result = {'modes': dict(modes_dict)}

# Print the result
print(yaml.dump(result, default_flow_style=False))

The result looks like this:

modes:
  off:
    pcod: pcod-off
    cwnc: cwnc-off
    influx: influx-off
  net:
    pcod: pcod-net
    cwnc: cwnc-net
    influx: influx-off
  net+influx:
    pcod: pcod-net
    cwnc: cwnc-net
    influx: influx-on
  net+file:
    pcod: pcod-net+file
    cwnc: cwnc-net+file
    influx: influx-off
  net+file+influx:
    pcod: pcod-net+file
    cwnc: cwnc-net+file
    influx: influx-on

This may even help with that WHOI request of no modes...

I'm not trying to derail any previous discussions but I think adding looping capability WITHIN the logger config deserves more thought/discussion.

This approach offers a lot of power while preserving valid YAML syntax and keeping the flexibility that most ships require. It also limits the code changes to just the template implementation.

As for looping, I would vote to do that upstream of the logger config like how the jinja implementation was done and even how my logger building script works.

This approach eliminates everything but the loggers section. The defaults objects and templates object can be moved to different files and added back in as includes. This makes building the loggers section via a pre-processing script easy... in fact maybe the jinja code should be brought to bear for that step.

Thoughts?

[davidpablocohn]

Innnteresting - thanks!

I'm not at all wedded to the looping thing - it just seemed like a handy shortcut.

I like the double-angle-brackets notation; obviates the need to have so many things quoted.

Not as keen on having each config say which modes it belongs to - that feels less intuitive and more error prone to me. But let me think on it a bit more?

Also, one minor note: in your configs: section, you're using a list of dicts, where I think you can just use dicts? (Unless I'm missing something)

  pcod:
    configs:
    - pcod-off: { modes: off }
    - pcod-net:
        modes: [net, net+influx]
        template:
          serial_to_net:
            kwargs:
              <<: *pcod_defaults
    - pcod-net+file:
        modes: [net+file, net+file+influx]
        template:
          serial_to_net+file:
            kwargs:
              <<: *pcod_defaults
              filepath: /var/tmp/log/pcod/raw/<<cruise>>_pcod

could be simplified as

  pcod:
    configs:
      pcod-off: { modes: off }
      pcod-net:
        modes: [net, net+influx]
        template:
          serial_to_net:
            kwargs:
              <<: *pcod_defaults
      pcod-net+file:
        modes: [net+file, net+file+influx]
        template:
          serial_to_net+file:
            kwargs:
              <<: *pcod_defaults
              filepath: /var/tmp/log/pcod/raw/<<cruise>>_pcod

right?

[webbpinner]

Moving from list to dict is absolutely the way to go as this would also prevent configs with duplicate names.

I agree with your comments on the modes. I was making a compromise between defining modes within the config vs it's own section (current model). In the end the current modes sections gets "built" from the configs so I'm guessing/hoping the logger_manager would first look for a modes section in the config and, if not found, would try to build it from the modes scalar/list within the config... and fail to load if neither is available.

[davidpablocohn]

I'm going to work on this bit by bit, first doing simple things like 1) allowing includes, 2) allowing configs to optionally be defined (not just declared) in the logger section, and 3) automatically creating a mode section if none is found.

But I've just come up with my first design question on includes :

If we have a section like

        includes:
          - included.yaml

what should it assume as a base directory? The install directory (/opt/openrvdas)? The directory from which the config file was loaded? Should we only allow full paths?

Should we maybe have another top level key, such as:

        includes_base: /opt/openrvdas/local/usap
        includes:
          - included.yaml

[yoLewshi]

I think assuming the installation directory with an option to override that through the key makes sense and should be flexible enough. It might also allow some standard "chunks" to be included in the repo (where one sample might "include" another file). Full paths is clearer but less flexible.