nanoFramework.IoT.Device dependency update process

[networkfusion]

Description

All repos that use a multi project solution for nugets (such as M5Stack) candidates and especially
nanoFramework.IoT.Device currently runs nuget dependency updates using GitHub workflows in 5 stages.
In each stage, the projects are stepped through and update sequentially (with optional exclusions).
This takes a very long time (especially stage 4)!

Once all affected projects are updated, this is re-ran by azure-pipelines before PR merge
and again after merge to main/develop.

We need to:

• work out how we can make this process as parallel as possible.
• remove duplicated builds.

Potential fixes:

• Should this be done in the repo (e.g. a single workflow yaml for each solution/project)?
• Should nanodu be able to parallelize the builds?
• Should dependency updates via GH workflow require the azure-pipeline check?
• Should we run the update checks on a Monday and Wednesday, rather than a Wednesday and Friday (when more core members are available)?

Note that there could/should be a catch all workflow to collect and process ones missing.

Which leads to (aims):

• Reduce build time (and associated costs).
• Update quicker (daily).
• Ensures users are not blocked by out of date nugets.

[torbacz]

I've develop simple tool to keep .net pacakges up to date, because DependaBot is not working with .net well. Should be pretty easy to integrate with NF system. The key advantage is that I'm not using nuget.exe and also I've implemented in memory cache for packages thus it's pretty fast - updating 10 SLN with ~10 project each, took ~2 minutes on azure pipeline. If you are interested, I could extend it with NF and github support.
https://github.com/torbacz/DependencyUpdated

On the other hand, there is ongoing .NET rework in DependaBot, no ETA yet.
dependabot/dependabot-core#10865

[josesimoes]

With the current workflow, I don't think we can't get past the exisiting "try to update" approach... Unless someone has a good idea on how to figure out if there is something to update. The current mechanism is already checking versions instead of performing the update, which was determined to be slightly quicker.

Having said that, the "obvious" performance upgrade to me seems to be firing parallel processes to tackle each repo.

I don't think there is a way out of the current staged run because of the interdependencies.

As for switching the the run schedule, nothing against it.

@torbacz the reason we went for a github action instead of azdo task was to take advantage of more availability simultaneously slots in github actions compared with limited azdo ones.
Still, haven't looked at your tool. Will do.
Question: anything "obvious" that you have there that could improve nanodu? 😉

[torbacz]

@josesimoes
The main differences:

• Nanodu is using nuget.exe tool, my tool is using REST API + XML parser (much faster)
• I've got InMemory Cachce for packages versions

[Ellerbach]

I've got InMemory Cachce for packages versions

you still have to build it once. Can we also expose it as a query table element? So anyone who would like to know which versions are related to which ones can have it?

[torbacz]

That's right. Durring build process, so first one reaches to nuget rest api to all versions and then logic is applied (update policies and others)
https://github.com/torbacz/DependencyUpdated/blob/b98ef8e8ec96159d80fe18a1dbfa5f4be1595578/src/DependencyUpdated.Core/Updater.cs#L175

In case of IoT.Device that would be great improvement - instead of reaching to NuGet server each time, you will have to do it only once per run.

To be honset, I would not expose it. There is already info in nuspec file about it, it can be checked there for each package.

[Ellerbach]

To be honset, I would not expose it. There is already info in nuspec file about it, it can be checked there for each package.

Yes, very fair!

[josesimoes]

nanodu and nanovc tools are in all our pipelines, very mature, tried and tested for a long time. As you know they are mostly wrappers around nuget CLI to serve the specificities of nanoFramework projects. I would rather improve what we have (preferably without introducing breaking changes) instead of switching to a new tool.
This is not to dismiss, by any means, your generous offer and the work that you've done there! 😃

Having said that, we could probably improve nanodu to use that approach of caching in memory the nuget details instead of using nuget CLI, if that's faster.
To be honest., I was under the impression that nuget CLI would be doing something similar with querying the local nuget cache instead of keep pounding the nuget server... but I never really tested it for that.