Yaml and Listings for more complicated objects

[dkyanakiev]

Hi,

While working on the yaml validation for alert rules(prometheus/vm) generated with pkl I ran into a yaml specific issue and I'm having trouble understanding the correct usage of listings for this usecase.

Valid syntax looks like this

groups:
  - name: 'service-alerts'
    rules:
    - alert: 'service-down'
      expr: 'count_metric < 3'
      for: '10m'
    - alert: 'service-two-down'
      expr: 'count_metric_two > 3'
      for: '10m'

I have the following structure

module basealert


// alerts: Listing<Alert>
class Alerts {
 name: String
 rules: Listing<SingleAlert>

 hidden def_chan: String
}

class SingleAlert {
 alert: String?
 expr: String?
 hidden duration: String?
 `for` = duration
 
 annotations: Mapping<String, String>
 labels: Mapping<String, String>

 
}



groups = new Alerts {
 rules {
   default {
     annotations {
       ["slack_channels"] = channelDefault
       ["slack_mentions"] = usersDefault
     }
   }
 }
}

hidden channelDefault: String = " "

hidden usersDefault: String = " "

hidden envName: String

And using this and amending it via

amends "basealert.pkl"
test {
    groups {
    // new {
        name = "test"
        rules {
            new {
            alert = "first-test-alert"
            expr = "if > 0"
            duration = "5m"
            annotations {
                ["alert_msg"] = "this alert is firing"
            }
            labels {
                ["team"] = "team-alfa"
                ["severity"] = "warning"
            }
            }   
            new {
            alert = "second-test-alert"
            expr = "my_metric > 1000"
            duration = "5m"
            annotations {
                ["alert_msg"] = "this alert is firing"
                ["slack_mentions"] = "@me"
                ["alert_group_description"] = """
                    long text description
                    """
            }
            labels {
                ["team"] = "team-alfa"
                ["severity"] = "warning"
            }
            }   
        }
        }
    }


channelDefault = "#basic_channel"
usersDefault = "@basic_mention"

Great.. but not really it generates result like

groups:
  name: test
  rules:
  - alert: first-test-alert
    expr: if > 0
    for: 5m
    annotations:

Which unfortunately is not valid as it needs to be a list type of obj to be validated by Prom/VM correctly..
I tried to shuffle the classes and force an additional objects but it would always conflict because I cant have dynamic object creation apparently.
I tried to create a new object and maybe copy over the properties like so

groups = new Alerts {
rules {
  default {
    annotations {
      ["slack_channels"] = channelDefault
      ["slack_mentions"] = usersDefault
    }
  }
}
}

grps {
  groups = new Listing{
    new {
      name = "test-outp"
      rules = groups[0].rules
    }
  }
}

But ran into

–– Pkl Error ––
A stack overflow occurred.

┌─ 9627 repetitions of:
│ 41 | rules = groups[0].rules

It must be something fairly basic I'm missing

[dkyanakiev]

Modifying the basealerts file to this

// alerts: Listing<Alert>
class Alerts {
  name: String
  rules: Listing<SingleAlert>

  hidden def_chan: String
}

class SingleAlert {
  alert: String?
  expr: String?
  hidden duration: String?
  `for` = duration
  
  annotations: Mapping<String, String>
  labels: Mapping<String, String>
 
  
}

class MyGroups {
  name: String
  rules: Listing<SingleAlert>
}



groups: Listing<MyGroups>

Solves the indentation and - name issue but then I lose the defaults

[odenix]

You only lost the defaults because you no longer specify a default for groups. Another problem I can spot is that some template (basealert.pkl) properties don’t have a type. For example, it should be `for`: String? = duration instead of `for` = duration and groups: Alerts = new {…} instead of groups = new Alerts {…}. If you don’t specify a type, template users can amend the template and set a different type, such as groups = "hello".

[dkyanakiev]

Those are nice suggestions indeed. I added them to the new version but ran into another issue. #388 (reply in thread)
I cant fit the defaults inside of the basealerts.pkl - It seems like I just cant wrap my head around how different types are instantiated and referenced.

[odenix]

I’m confused. Your very first basealerts.pkl was setting a default for groups. Why did you stop doing this?

[dkyanakiev]

I couldn't figure out how to do it inside here

groups: Listing<Alerts> = new{
}

With this syntax and object setup.. The original setup did not render the entire object the way I wanted it to

[odenix]

Setting a default for groups is certainly the right direction, and setting a default for Listing is very similar to setting a default for Mapping. Writing a good template isn’t trivial, especially if you want things like overridable defaults.

[holzensp]

Which unfortunately is not valid as it needs to be a list type of obj to be validated by Prom/VM correctly..

The "it" in that sentence refers to groups? Your problem is that you define

groups = new Alerts {

There are two things "wrong" with that;

1. it's not a Listing<Alerts> (btw, s/Alerts/Alert, because the class expresses a single one and that's what confused you, if I were to guess)
2. it doesn't actually define a type for groups (your code means the same as groups: unknown = new Alerts {).

If I understand your scenario correctly, what you want is

groups: Listing<Alert> = new {
  new {
    rules = ...

[dkyanakiev]

the "it" refers to the the generated object (elements of group). It should have been like this, to be validated correctly.

groups:
  - name: test
     rules:
     ......

What I ended up having currently is

module basealert


class Alerts {
  name: String
  rules: Listing<SingleAlert>

  hidden def_chan: String
}

class SingleAlert {
  alert: String?
  expr: String?
  hidden duration: String?
  `for`: String? = duration
  
  annotations: Mapping<String, String>
  labels: Mapping<String, String>
 
}

class MyGroups {
  name: String
  rules: Listing<SingleAlert>

}




groups: Listing<Alerts> = new{
}


hidden channelDefault: String = " "

hidden usersDefault: String = " "

hidden envName: String

output {
  files {
    ["\(envName)-results.yaml"] {
      value = groups
      renderer = new YamlRenderer {}
    }
  }
}

and my reference

amends "basealert.pkl"


groups {
    new {
        name= "x"
        rules {
            default {
                annotations {
                    ["slack_channels"] = channelDefault
                    ["slack_mentions"] = usersDefault
                }
            }
            new {
                alert = "first-test-alert"
                expr = "if > 0"
                duration = "5m"
                annotations {
                    ["alert_msg"] = "this alert is firing"
                }
                labels {
                    ["team"] = "team-alfa"
                    ["severity"] = "warning"
                }
            }   
            new {
                alert = "second-test-alert"
                expr = "my_metric > 1000"
                duration = "5m"
                annotations {
                    ["alert_msg"] = "this alert is firing"
                    ["slack_mentions"] = "@me"
                    ["alert_group_description"] = """
                        long text description
                        """
                }
            labels {
                    ["team"] = "team-alfa"
                    ["severity"] = "warning"
                }
            }   
        }
    }
}


channelDefault = "#basic_channel"
usersDefault = "@basic_mention"
envName = "local-env"

Currently this solves some problems but also creates others.

1. Because of the output files. It deletes the top level when outputting to file I had to wrap it in 1 layer on top

alert_module = new {
  groups = new Listing{
  }
}

However trying to wrap it inside doesn't work.

alert_module {
groups: Listing<Alerts> = new{
}
}

If I drop the output to file, it renders everything as it should but directing to file is the problem.

2. Setting defaults inside my module (basealert.pkl) vs at the reference level. Minor issue but it shouldn't be impossible right?

[holzensp]

Why are you using output.files if you only ever want the single file? Is that just to control the file name?

As to your questions:

1. If you look at the default definition of output.value (here) you see that it's the module it's in. In the case of multi-file output, that default doesn't necessarily make sense and so there is no default for value either in the FileOutput class (here) or in output.files (here). If you still just want the complete module, instead of value = groups, you say value = module.
2. There is an important difference between setting defaults in a class definition or in class usage (and every module you define, implicitly defines its class). @HT154 wrote a very complete demonstration of these differences here. Your defaults for channelDefault and userDefault look pretty suspicious to me. I'm guessing you only have those, because you wanted to use them in groups.rules.default. You don't have to; you can refer to a property before any value is provided.

[dkyanakiev]

Why are you using output.files if you only ever want the single file? Is that just to control the file name?
Its actually because the idea for this is to use it in bit larger scale
something like this

output {
  files {
    ["\(envName)/alert-rules-\(teamName)-alert-\(moduleName).yaml"] {
      value = module
      renderer = new YamlRenderer {}
    }
  }
}

Would probably be a better representation of what the end state would be. Since it would render different files for different environments to be consumed in my pipelines. And actually yes.. the output = module seems like exactly what I needed. Sadly couldn't find a reference about it in the docs, have to look at source code a bit more I guess.

2. Thanks a lot for that link. I would give it a proper read and hopefully will be able understand things better.
   I think my understanding of the class definitions and how it should look like are ok but my implementation is lacking.
   To give more context to those defaults.
   If we look at the SingleAlert class

class SingleAlert {
  alert: String?
  expr: String?
  hidden duration: String?
  `for` = duration
  
  annotations: Mapping<String, String>
  labels: Mapping<String, String>
  
}

My goal was to have a set of defaults (keys) for annotations and labels - that will and should always exist and also allow for additional/optional ones. Their values however would change depending on what is provided inside the files that references the.
I couldn't get there unfortunately so the next best thing was the usage of defaults - thanks to @translatenix who showed me how to implement it
I tried variations of

 annotations = new Mapping { 
    ["channel"] = channelDefault
    ["users"] = usersDefault
  }

but would error out like this

–– Pkl Error ––
Cannot reference property `channelDefault` from here because it is not `const`.

19 | ["channel"] = channelDefault
                   ^^^^^^^^^^^^^^
at basealert#SingleAlert.annotations["channel"] (file:///Users/dkyanakiev/Workspace/gitlab/pkl-rick/basealert.pkl, line 19)

To fix, do either of:
1. Add modifier `const` to property `channelDefault`
2. Self-import this module, and reference this property from the import.

[holzensp]

Ah, yes; default values for properties inside class definitions must be const or refer to members of the class. Otherwise, amending the module changes the meaning of the classes it defines.

So, if I understand you correctly, you're trying to force the user to provide values for those keys. You can simply constrain the type of the mapping to say that;

annotations: Mapping<String, String>(containsKey("channel"), containsKey("users"))

If the number of keys grows and you want to give users nice autocomplete, you can also define an enum;

typealias AnnotationKey = "channel"|"users"|"and"|"many"|"more" // | ... | ... | ...

// This is such a common pattern, it should probably be a stdlib function.
local allAnnotationKeys = (reflect.TypeAlias(AnnotationKey).referent as reflect.UnionType).members.map((it) -> it.value)

annotations: Mapping<AnnotationKey|String, String>(allAnnotationKeys.every((it) -> containsKey(it))