Specify that a field can be one of a list of options

[eivindjahren]

Lets say you have the following kind of specification:

vehicle_list:
  - spaceship:
      name: "Millenium Falcon"
      crew:
         - "Han Solo"
         - "Chewbacca"
  - time_machine:
     name: "The Dolorean"
     inventor: "Doc Brown"
  - spaceship:
     name: "The Enterprise"
     crew:
         - "Jean-Luc Picard"
         - "Data"

It would be nice to have a OneOf combinator to specify that the node matches one of a known list of schemas:

schema = {
    MK.Type: types.NamedDict,
    MK.Content: {
        "vehicles_list": {
            MK.Type: types.List,
            MK.Content: {
                MK.Item: OneOf([
                     {MK.Type: types.NamedDict, Mk.Content: {"spaceship": {}},
                     {MK.Type: types.NamedDict, Mk.Content: {"time_machine": {}},
                ])
            }
        }
    }
}

This would also be handy for specifying that a field can be either a string or a float:

OneOf([{MK.Type: types.Float}, {MK:Type: types.String}])

[lars-petter-hauge]

Currently if this is needed a possible solution is to create validators that can check if the content is one of a set of items. But that does not work for containers of different structure, nor does it go well with documentation generation.

[markusdregi]

This will make the complexity of the data skyrocket and violate a design principle of configsuite, namely that all data fields are known up front. In other words, if vehicle_list[0].spaceship exists, then so should vehicle_list[1].spaceship. This complexity will in my opinion hit the programmer, but more importantly communication with the users.

I think the better solution in your concrete example would be to introduce a (vehicle_)type that can be either spaceship or time_machine and then implement specific validators for the two. If this turns out too complicated, then I think that is a clear hint that perhaps the users should not be exposed to a vehicle_list, but a spaceship and a time_machine list 🤷

[eivindjahren]

all data fields are known up front [...] if vehicle_list[0].spaceship exists, then so should vehicle_list[1].spaceship.

I am not sure how to apply the design principle in terms of lists, but would it be correct to say "All lists must be homogenous"?

I think the better solution in your concrete example would be to introduce a (vehicle_)type that can be either spaceship or time_machine

In this case the vehicle_type is the type OneOf(spaceship, time_machine) and gets the job done. But there is no generation of the documentation of the fields of spaceship and time_machine. I guess this could be resolved by providing an interface for documentation of custom types, so that configsuite.docs.generate does not compare type, as it is now, but instead calls a method on the type in order to generate the doc. Then types own how to generate its documentation:

configsuite/configsuite/docs/__init__.py

Lines 83 to 84 in 847e4bc

	elif isinstance(schema[MK.Type], types.BasicType):
	docs += [indent + ":type: {_type}".format(_type=schema[MK.Type].name)]

but instead calls the generate_doc() of the type:

Type = collections.namedtuple("Type", ["name", "validate"])


class BasicType(Type):
   def get_docs(self, indent, schema):
      return  [indent + ":type: {_type}".format(_type=self.name)]


class NamedDict(Type):
   def get_docs(self, indent, schema):
      def req_child_marker(key):
            child_req = not (
                schema[MK.Content][key].get(MK.AllowNone, False)
                or schema[MK.Content][key].get(MK.Default, None) is not None
            )
            return "*" if child_req else ""

        return [
            "\n".join(
                (
                    indent
                    + "**{key}{req}:**".format(key=key, req=req_child_marker(key)),
                    value[MK.Type].generate_doc(level=level + 1, value),
                )
            )
            for key, value in schema[MK.Content].items()
        ]

def generate(schema, level=0):
   indent = level * 4 * " "
   element_sep = "\n\n"

   docs = [indent + schema.get(MK.Description, "")]
   docs += schema[MK.Type].generate_doc(indent, schema)
   return element_sep.join(docs)

This solves the issue of documentation, but such a type cannot be recursivly transformed over, be defaulted, and contained
NamedDict are just validated to be dicts, not contain known fields, although the same refactoring as above could
be made to allow types to have their own behavior for those cases as well.

then I think that is a clear hint that perhaps the users should not be exposed to a vehicle_list, but a spaceship and a time_machine list

I guess this could work:

spaceships:
  -  name: "Millenium Falcon"
     crew:
         - "Han Solo"
         - "Chewbacca"
  - name: "The Enterprise"
    crew:
         - "Jean-Luc Picard"
         - "Data"
time_machines:
     - name: "The Dolorean"
       inventor: "Doc Brown"

Here the lists in time_machines and spaceships are defaulted to zero and the fields are optional, giving
identical behavior except for location of lists.

It becomes a bit of a mind twist in the following example but I guess it is not too bad:

coctail:
   - rum:
       brand: "Havana Club"
       fraction: 0.5
  - milk:
      fat_content: 1 # percent
      kind: "bovine"
      fraction: 0.5

transforming to homogenous lists you get:

coctail:
   rums:
      - brand: "Havana Club"
        fraction: 0.5
  milks:
      - fat_content: 1 # percent
        kind: "bovine"
        fraction: 0.5

[markusdregi]

I am not sure how to apply the design principle in terms of lists, but would it be correct to say "All lists must be homogenous"?

All containers of arbitrary size needs to be homogenous. List elements must be homogenous. Furthermore, the keys in a dict must be homogenous and the dict values must be homogenous. It is only the named dicts, where you specify both all the keys and which type each of them maps to up front, that allows for different types within the same structure :)

[markusdregi]

I think the idea of deferring the documentation to the type is probably a good idea 👍 But as you say, transformations, defaulting etc will be difficult with the multi-type suggestion of this issue 🤷

[markusdregi]

In the cocktail example I'm not sure how much you expect to know up front about the ingredients. If there are certain elements you expect to be present for each of them and some that are arbitrary metadata you can do:

coctail:
   -
     category: rum
     brand: "Havana Club"
     fraction: 0.5
  -
    category: milk
    brand: Happy Cow
    fraction: 0.5
    additional_info:
      fat_content: 1 # percent
      kind: "bovine"

Where you can add validators about certain types of additional_info to be present for various categories if you want to.

I also notice that you have a strong preference for making the list elements into a dictionary where the first level is a single key representing one of the data elements, like:

   - rum:
       brand: "Havana Club"
       fraction: 0.5

I would find it really interesting if your users find that easier to read then the above category example. However, notice also that a layered transformation can be used to completely transform the data layout. But, the complexity goes up of course :)