[FEATURE] KiRi integration

[FrankvVeelen]

As an alternative to the kidiff tool (which works okay, but doesn't work for analyzing differences between more than a few commits/versions, I think it'd be cool if kibot could integrate Kiri, as it (at least for me) works a bit nicer with tools like jenkins (generating a html page on which i can then analyse differences between commits).

If you need help with implementing this and could give me a few pointers where i should start please let me know.

[set-soft]

When I took a look at Kiri it used a mechanism for schematics that isn't good for KiBot integration.
I know Kiri is much better for interactive usage, but not for CI/CD.
I'll keep the request, but I'm not sure about it.

[Nudelsalad]

Would be an awesome feature! Does Kiri even interfere with kibot if it is triggered separately? With the --no-server flag given the output could be maybe nicely relinked with navigate_results maybe?;) There are workarounds to get this working but up to your answer I guess there is nothing in the pot in the near future?

[set-soft]

Hi @Nudelsalad !

I see @leoheck moved to kicad-cli for KiCad 7. Which removes the need for PlotKiCadSch, a real overkill since this is an OCaml tool.

Still using KiCad-Diff for the PCB, which is redundant tool since KiBot uses KiDiff.

Things are getting closer, but I don't have time right now.

[leoheck]

Hi guys, KiCad-Diff is a small script, and for Kiri, you don't even need the GUI, I don't see any big problem of having it installed.

Kiri still uses KiCad-Diff because kicad-cli for layouts takes a looong time to export each layer, and considering a big set of layers per commit and also a big set of commits, it takes a lot of time.

But it is also possible to use kicad-cli for this. If KiCad-Diff is not installed (or maybe you have to pass a flag to Kiri), it will use kicad-cli instead. It may work ok if comparing 2 commits.

For instance, here, we just compare 2 commits, removing previous runs, and displaying the commands kiri is running, without launching the browser.

kiri -r -D --kicad-cli -t 2 -l 

And this already takes 12.2s on my machine which is a good machine I guess (Dell XPS, i9)
The Board used in this example has 22 layers. Not all of them are important but they are/were present in the Kicad project.

[set-soft]

Hi @leoheck !

I'll take a look at this. I'll post my questions here. First one: why dos2unix is a mandatory dependency?

[leoheck]

Nope, I just ended up adding it as a workaround since most of the time people do things that they don't know, and they blame my tool for that and I don't always have time to debug everything. You could remove the lines using it. I can end up removing or adding a test to not use it if it is missing.

[set-soft]

Would be an awesome feature! Does Kiri even interfere with kibot if it is triggered separately? With the --no-server flag given the output could be maybe nicely relinked with navigate_results maybe?;) There are workarounds to get this working but up to your answer I guess there is nothing in the pot in the near future?

About the link to navigate_results, you mention "workarounds", which ones do you mean?

From what I see you need to start the KiRi webserver and read the pages from it. As a workaround I tried google-chrome --allow-file-access-from-files index.html and got it working, but is quite specific to a particular case.

[leoheck]

I am not sure how KiBot works or how it is used, but Kiri could just generate the images and the website without having to launch it... then the result (the .kriri folder,) can be exported, there is also an argument to make a pack of it. Inside, this folder it has a python script to launch the webserver using the previously generated files.

[set-soft]

@leoheck : I was playing with KiRi and found a couple of issues:

1. When the files are in KiCad 5 format it tries to use plotkicadsch, even when it used kicad-cli for the other commits (forced by --kicad-cli). You can see it using the following repo
2. In the above case when plotkicadsch fails (not installed) it tries to use xdootool. But if the process fails the keyboard is left changed to us. I suspect one of the problems is that KiRi thinks this is KiCad 5, but started KiCad 7.

It would be nice if --kicad-cli forces the use of kicad-cli even for KiCad 5 files.

Other questions:

• I see you use shell scripts and Python, why not just Python? Is much more powerful and is even more portable, you won't need to worry for things like readlink, sed, realpath, etc. And your shell scripts are Bash specific!
• By using something like Python you'll be able to drastically reduce the "bin" pollution, you have 19 things in bin, KiRi can't be installed in a regular system, in particular: you even have an env.sh script, that could collide with anything.

[set-soft]

I am not sure how KiBot works or how it is used, but Kiri could just generate the images and the website without having to launch it... then the result (the .kriri folder,) can be exported, there is also an argument to make a pack of it. Inside, this folder it has a python script to launch the webserver using the previously generated files.

One of the KiBot outputs can generate web pages to browse all the generated stuff. The problem here is that we can't directly link the KiRi output from this page.

From what I understand the big problem is that KiRi needs to read contents from separated files and this seems to be forbidden for Java Script. Embedding all the SVG files seems to be an overkill and this is why a server is needed.

[leoheck]

It would be nice if --kicad-cli forces the use of kicad-cli even for KiCad 5 files.

Yeah, definitely, I will give it a look.

I see you use shell scripts and Python, why not just Python? Is much more powerful and is even more portable, you won't need to worry for things like readlink, sed, realpath, etc. And your shell scripts are Bash specific!
By using something like Python you'll be able to drastically reduce the "bin" pollution, you have 19 things in bin, KiRi can't be installed in a regular system, in particular: you even have an env.sh script, that could collide with anything.

Yeah, python is awesome, but shell script is easier for me. Actually, this tool started as an experiment, but then later I adapted it to work on other systems and for others too. It just started as a bunch of commands that I put together in a script. Having everything made in Python would be better definitely it would also make it possible to add on Kicad package manager (people would definitely love this). I thought many times to start changing it to Python only, but it would take a lot of time and effort for me to do it. Maybe if I have some spare time in the future. It is not a big and complex tool. The tool is simple, actually, it gets the commit list from git and iterates through them exporting the schematics/layers to images.

[set-soft]

Hi @leoheck !

I tried the following toy test:
t1.zip

Running the run_kiri.sh script I got a nice list of commits and can browse the schematic changes. But the PCB looks the same for all the commits. Looking at the logs I see things like:

# 4/6 1db11f8 | 2023-11-27 11:12:45 | KiBot test | Reference

    Retrieving commits:
    - Source path: /home/salvador/0Data/0Test/#391/t1/.
    - Output path: /home/salvador/0Data/0Test/#391/t1/.kiri/1db11f8
...
    # Plotting schematics 1db11f8
...
   # Plotting layout _local_, F.Cu

Always plotting local, not the corresponding commit. Can you reproduce this bug?

[leoheck]

Schematics are hard to spot the differences actually.

But I can see them here...

You can easy see the same with this script

schs=($(find .kiri/*/_KIRI_/sch -name "*.svg"))
for sch in ${schs[@]}; do
        # echo "Diffing ${schs[1]} and ${sch}"
        diff -q --report-identical-files ${schs[1]} ${sch}
done

Right now, when the local files have not commited changes, they are included automatically on the list of commits.