Add possibility to include multiple configs

[bugy]

Hello there!
Is there any way of including more than one external included file for script? thinking of one of the following:

• Multiple include directives (I think it only works the latest)
• The possibility of an include directive with array as a type.
• Another include on the included file.

I'm trying to figure out a way of having configuration files that will be the same across installations, some specific to each one and others specific to the servers.
Is there any way of achiving this in the current version? will you consider implementing something like this in the future?

Thanks in advance.

Originally posted by @kbza in #133 (comment)

[bugy]

@kbza I moved the conversation to a new ticket.

[bugy]

So, if you would like to implement it, you would need to change src/model/script_config.py file.

First of all, to reduce the work for you, here is the modification in src/model/model_helper.py, which you would need:

def fill_parameter_values(parameter_configs, template, values, *, support_arrays=False):
    result_list = [template]

    for parameter_config in parameter_configs:
        if parameter_config.secure or parameter_config.no_value:
            continue

        parameter_name = parameter_config.name
        value = values.get(parameter_name)

        if value is None:
            value = ''

        if support_arrays and isinstance(value, list) and (parameter_config.type == PARAM_TYPE_MULTISELECT):
            mapped_list = parameter_config.map_to_script(value)
            new_list = []
            for mapped_value in mapped_list:
                for result in result_list:
                    new_list.append(result.replace('${' + parameter_name + '}', str(mapped_value)))
            result_list = new_list
            continue

        if not isinstance(value, str):
            mapped_value = parameter_config.map_to_script(value)
            value = parameter_config.to_script_args(mapped_value)

        result_list = [result.replace('${' + parameter_name + '}', str(value)) for result in result_list]

    if len(result_list) == 1:
        return result_list[0]

    return result_list

After that, in script_config.py:338, replace that line with:
self.value = fill_parameter_values(self._parameters, self._template, self._values, support_arrays=True)
(like by adding this parameter , support_arrays=True)

After that you would need to adjust _path_to_json method. At the moment it accepts path parameter. With the change above, this parameter can be either a string (for a single path) or a list (for multiple paths).
So, if it's a list, I would suggest to iterate over it, call _path_to_json for every element and then merge resulting dicts.

[kbza]

Hello there,

I've managed to implement the solution you proposed (actually it was almost fully done by you). Practically, the only change was to merge the dicts recursively in case of path being a list inside _path_to_json:

        if not isinstance(path, str):
            merged_dict={}
            for path_item in path:
                if 'parameters' in merged_dict:
                    merged_dict['parameters']+=(self._path_to_json(path_item))['parameters']
                else:
                    merged_dict.update(self._path_to_json(path_item))
            print(merged_dict)
            return merged_dict

There was a caveat that I think is related to the way I've defined the script conf and this particular implementation:

{
  "name": "Param Test",
  "script_path": "/home/platform/script-server/scripts/paramTest.sh",
  "include": "${Server Select}",
  "parameters": [
    {
      "name": "Server Type",
      "type": "multiselect",
      "values": [ "app", "db", "web", "lts" ],
      "separator": "|"
    },
    {
      "name": "Server Select",
      "type": "multiselect",
      "values": { "script": "find /home/platform/script-server/conf/platform_files/ -regex '.*[${Server Type}]+[0-9]+_server.json' -type f"},
      "separator": ","
    }
  ]
}

When I select more than 1 parameter in the first multiselect ("Server Type"), it tries to load every selected file for each one of the types selected. It works but warns me everytime:
2020-09-06 20:32:26,786 [script_server.script_config.WARNING] Parameter IP app101 exists in original and included file. This is now allowed! Included parameter is ignored
I've managed to "remove duplicates" on fill_parameter_values with this dirty trick:
result_list = list(dict.fromkeys(result_list))
but I don't think this is a very good approach... this is more a lazy patch.

This works though... I think this is a very specific solution that only works for this use-case (using multiselects).

For me, the final solution would be:

• Keeping everything that include directive can do now (one file, use of templates).
• Add support for multiselect as a list of files
• Add support to specify a list of files as an array
• Add possibility to mix everything of the above
• Extra: a server-conf defined "env_var" as a common root directory for include files.

I've kinda made a config like this work:

{
  "name": "Param Test",
  "script_path": "/home/platform/script-server/scripts/paramTest.sh",
  "include": [ "/home/platform/script-server/conf/platform_files/bssbi101_server.json", "${Server Select}"],
...
}

But:

• It only works as expected if something is selected in "${Server Select}"
• I don't think the changes I've made in code are the correct answer for this. I would like some advice in how to structure this not to mess up the code very much.

But why all this trouble?
The ultimate target will be to deploy this application in several different sites with a minimum os changes between them, so:

• Server configuration will be prety similar on new sites.
• Scripts will be exactly the same among them.
• Some of this include files will be common, shared among sites.
• Some other, will be site dependant. Examples of this could be networks, IPs, paths, OS dependant variables... This will be filled in by sysadmins on initial site configuration.
• All this files will contain hidden and const parameters. Support users don't need to be aware of these env_vars.
• Scripts will be implemented using these constants (PARAM_... approach) whenever necessary.

I think this changes will improve the portability and will simplify configuration once structured (at least for those of us that plan to install this software dozens of times ;) ).

[bugy]

Hi @kbza, to be honest, I'm not fully understand your proposal :)

Extra: a server-conf defined "env_var" as a common root directory for include files.

Could you give an example, please?

Add support for multiselect as a list of files

This sounds fine to me. I had an impression, that you only need multiselect support.
I guess the main implementation problem here would be to merge lists, but it's doable. If you want, I can check your repo, if you already pushed the changes

it tries to load every selected file for each one of the types selected. It works but warns me everytime:

If you could share your code and configuration examples (you can prepare just some test files), I can try to debug and check it

[kbza]

Extra: a server-conf defined "env_var" as a common root directory for include files.

I war refering to something like "${auth.username}" that can be used in scripts, but this one will be something like "${include_path}". If needed, you may configure it as a server configuration property and use it to refer a common path were every include file will be placed (instead of using the hole path for every file).

Add support for multiselect as a list of files

This is the one thing we've already achieved with the changes you proposed. The problem is, that I mostly don't want to offer the user the choice to select include files.
On most of the scripts, this include files will be included literally by the one person that knows that is needed (the one that develops the script). For example, imagine you are creating a script that will need to access one particular database, connect to a set of servers to do some action and send an email using the platform configured SMTP server... this script will be always the same among sites/platforms, it will load SMTP_config.json, application_servers.json, and database_conf.json; and this are the files that will change.
Same "code", but adapted to site based only on these "included configuration files".
A workaround could be a "fixed" on constant multiselect... if that makes any sense.

If you could share your code and configuration examples (you can prepare just some test files), I can try to debug and check it

You can check the canges here: https://github.com/kbza/script-server/tree/multi-include

• samples/configs/include_files: you can see some server config examples.
• samples/configs/Param_Test.json: the runnable.
• samples/scripts/paramTest.sh: the script to test the loaded parameters/variables

of course, this will be refactored prior to any pull request... for now this is just a concept.

[bugy]

Hi @kbza unfortunately I didn't have time so far to check your code. Will try to do it tomorrow late evening

[bugy]

it tries to load every selected file for each one of the types selected. It works but warns me everytime:

My bad there, I made wrong implementation for variable substitution. Here is the updated method:

def fill_parameter_values(parameter_configs, template, values, *, support_arrays=False):
    if isinstance(template, list):
        result_list = copy.copy(template)
    else:
        result_list = [template]

    for parameter_config in parameter_configs:
        if parameter_config.secure or parameter_config.no_value:
            continue

        parameter_name = parameter_config.name
        value = values.get(parameter_name)

        if value is None:
            value = ''

        if support_arrays and isinstance(value, list):
            mapped_list = parameter_config.map_to_script(value)
            new_list = []

            if len(result_list) == 1:
                result_list = result_list[0]
            result_list = list(dict.fromkeys(result_list))

            for result in result_list:
                something_replaced = False
                for mapped_value in mapped_list:
                    replaced_value = result.replace('${' + parameter_name + '}', str(mapped_value))
                    if replaced_value != result:
                        new_list.append(replaced_value)
                        something_replaced = True

                if not something_replaced:
                    new_list.append(result)
                    
            result_list = new_list
            continue

        if not isinstance(value, str):
            mapped_value = parameter_config.map_to_script(value)
            value = parameter_config.to_script_args(mapped_value)

        result_list = [result.replace('${' + parameter_name + '}', str(value)) for result in result_list]

    if len(result_list) == 1:
        return result_list[0]

    return result_list

I would like some advice in how to structure this not to mess up the code very much.

Regarding this concern, I think your idea is fine. We can just fine-tune the implementation. For example, when you search for required_parameters inside TemplateProperty, you can extract common logic to a method and then the corresponding part in the init method would look smth like:

        if isinstance(template, list):
            for template_element in template:
                required_parameters.update( find_required_parameters(template_element) )
         elif template:
                required_parameters.update( find_required_parameters(template) )

Where find_required_parameters would be a new function (containing this while loop)

One more feature, which you could add, is support for environment variables, there is a method: model.model_helper.resolve_env_vars. If you add a usage for it (for example after you call config_object.get('include') in script_config), then you would be able to have smth like:
include: [ "$$MY_ENV/abc.json", "$$HOME/${Server Type}"]

[bugy]

Will be addressed in #423

[bugy]

Done, now "included" option accepts an array. In case of a conflict (same options in different included files), the first defined included file takes precedence.