Request for an alternative to supporting dynamic imports - need a way to pass a reference path (library path?)

[n-kremeris]

Hello all, and sorry for resurrecting the dynamic include topic again. I appreciate this has been answered numerous times, and i see there is at least one way to approximate dynamic includes, but this doesn't effectively solve the issue i have at hand.

I'm an extensive user of Jsonnet, i'm using it to generate very complex configurations via many levels of "import"s, allowing me to have extremely short top-level .jsonnet files, promoting great configuration code reuse. For reference, i'm rendering the Jsonnet in python, and using the latest v0.20.0 version.

The problem i'm facing is in using jsonnet together with jenkins build pipelines, where the job folder path is dynamic, and i'm using two separate separate repositories in this single pipeline (and said repositories absolutely must stay separate). To allow parallelism on jenkins (building the same pipeline with different revisions), we have adopted a "job folder" naming, which i call the base path, as /.../<job_name>/<job_id>/. This is unique per every job.

I'm using jsonnet in both repositories, lets call them A at /.../<job_name>/<job_id>/A and B at /.../<job_name>/<job_id>/B. at The B repository has the top level .jsonnet files alongside many other .jsonnets that get imported via relative paths, so this is all good. However, i additionally need to import some configuration files from A, for example, /.../<job_name>/<job_id>/A/cfg1.jsonnet, and cfg2, cfg3 - cfgN . It's obvious that this import name isnt something that's needs to be created "dynamically", e.g. the file name and path relative to the base path is fixed, but the base path changes with every run, and there is seemingly no way to do local cfg1 = import ref_path + "cfg1.jsonnet".

I've considered numerous solutions to this problem:

• local relative_path = "/.../<job_name>/<job_id>/A/"; local cfg1 = import relative_path + "cfg1.jsonnet" - this is the ideal solution, obviously doesn't work due to jsonnet computed imports are not allowed
• Hardcoding a relative path from B to A like ../A/cfg1.jsonnet - not possible in my case, because the name A changes between different types of pipelines, and therefore it cannot be hardcoded.
• Importing configs via ext_codes, e.g. ext_codes = { cfg1 : "/.../<job_name>/<job_id>/A/cfg1.jsonnet, cfg2:..., cfg3:..."} - not possible either because i may have two sets of A repositories in use, e.g. /.../<job_name>/<job_id>/A1, /.../<job_name>/<job_id>/A/, and additionally, it forces passing these arguments every time, but there are cases where the top level .jsonnet doesn't include anything from A at all (meaning i would have to have at minimum two different calls to the jsonnet.evaluate_file, AND it would require me to know if A is in use or not, further complicating the jsonnet rendering process)
• handling the entirety of A configs separately from B and merging them together in the backend - this is what is currently done, and i'm writing this post specifically for help to move away as its not flexible enough for my needs. I want to be able to dynamically modify and use the configurations from A inside B, e.g. take one parameter from cfg1 and sweep it, creating multiple copies of cfg1, in a .jsonnet file in B

I also have an additional complexity of there being a A/reshaper.jsonnet with a function in it that is to be included in .jsonnet files from B, and will take the config data from A.

For a more realistic example of what i'm trying to do, here is an extremely short version of what i'm trying to accomplish.

• /common_path/A1/cfg1.jsonnet

{
    name: "cfg1",
    size: 1
}

• /common_path/A1/reshaper.jsonnet

local reshape(data) = [
   "--" + idx.key + "=" + idx.value
   for idx in std.objectKeysValues(data)
]
{
    reshape: reshape
}

• similarly, duplicate the above for /comon_path/A2/...

• /common_path/B/top_X.jsonnet

local cfg1 = import std.extVal("common_path") + "A1/cfg1.jsonnet"
local cfg2 = import std.extVal("common_path") + "A1/cfg2.jsonnet"
local cfgN = import std.extVal("common_path") + "A1/cfgN.jsonnet"
local reshaper = import std.extVal("common_path") + "A1/reshaper"
local reshape = reshaper.reshape

local cfg1_modified = ...
local cfg2_modified = ...
local cfgN_modified = ...

{
   "banana" : 1,
   "apple" : 2,
   "cfgs" : [reshape(cfg1), reshape(cfg2)...reshape(cfgN)]
}

• /common_path/B/top_Y.jsonnet

local cfg1 = import std.extVal("common_path") + "A2/cfg1.jsonnet"
local reshaper = import std.extVal("common_path") + "A2/reshaper.jsonnet"
<similar to top_X.jsonnet>
{}

• /common_path/B/top_Z.jsonnet

<doesn't include anything from A1 or A2>
{}

• render as follows:
  • jsonnet.evaluate_file("<..>/top_X.jsonnet", ext_vars="{common_path : "/common_path/}") - uses the common_path
  • jsonnet.evaluate_file("<..>/top_Y.jsonnet", ext_vars="{common_path : "/common_path/}") - uses the common_path
  • jsonnet.evaluate_file("<..>/top_Z.jsonnet", ext_vars="{common_path : "/common_path/}") - ignores the common_path but still works fine

Before you say it... yes, i am doing something horribly complex, and thankfully jsonnet so far has been able to cope with absolutely everything i throw at it so far, except for this current common path problem i'm trying to solve. (To give you a sense of scale here, one of the top_X.jsonnet files in my use cases is 51 lines long, and renders into json that's >3000 lines long).

I hope i'm just missing something very obvious and simple here, something like an way to pass a library path and import files relative to this library path.

[johnbartholomew]

Edit: This is unnecessarily complicated (though it's pretty flexible if you're doing something even more fiddly). See my second comment below for an easier way.

If you're using the Jsonnet Python module, have you tried implementing a custom import_callback function?

Unfortunately it is not very well documented, though its existence is mentioned here: https://jsonnet.org/ref/bindings.html#python_api and it is used in the Python test: https://github.com/google/jsonnet/blob/master/python/_jsonnet_test.py

Here's an example: https://gist.github.com/johnbartholomew/98ea7e04081f762aecb6fbacd7a82b52

The import_callback receives two arguments: the directory of the file doing the import, and the string name/path that was requested to be imported. You can put whatever logic you want to decide what to do with those values, and return two results, the path of the file you found to satisfy the import, and the content of that file.

You can use this to implement a variety of weird things like loading imports from the internet or generating them dynamically on demand somehow, but you can also use it fairly simply to implement a search path type lookup system for imports.

[johnbartholomew]

Oh, actually if you just need a search path there's a much easier way (silly of me not to check this first). The jpathdir keyword argument lets you specify a list of search paths to check for imports. I've added use of jpathdir to the example gist.

[n-kremeris]

Thanks John for the reply, and i really appreciate the additional example you uploaded, i'm sure i'll find it handy!

Since opening this issue, the native callbacks and import callbacks were what i've been looking at. But it seems that defining a custom import callback does not bypass the "computed imports not allowed" problem. I think what i should have mentioned additionally in the original post was that the "common_path" is used for more than just the configs - during evaluation, i'm generating full paths to various things, e.g. something like common_path + "A/bin/xxx.elf or common_path + "A/misc/bootloader.bin.

Given your comment, I now see i could do something like local cfg1 = import '__COMMONPATH__/A/cfg1.jsonnet and then handle the replacement/search in the custom import_callback implementation, however, that means that i will now have the same path fragment "encoded" in two different ways. What i mean is this

• in python, i pass the common path as an argument, e.g. --common_path "/common_path"
• currently, i pass it in and use it in jsonnet via extVars = {"common_path" : args.common_path},
• will need to use the same args.common_path to dynamically replace "COMMONPATH" in the import_callback

The above is likely a suitable, albeit quite an ugly workaround (provided i don't run into more issues). It would basically lead to the following approximately:

local cfg1 = import "__COMMONPATH__/A/cfg1.jsonnet";
local reshape = (import "__COMMONPATH__/A/reshaper.jsonnet").reshape;
local misc = import "./lib/misc.jsonnet";
local common_path = std.extVars("common_path");

{
   <...>,
   example_bin : common_path + "A/bin/xxx.elf```
}

Unless a nicer solution magically presents itself, i guess i'll just have to use the above :)