Inlined dependencies in mip packages

[aivarannamaa]

According to https://github.com/micropython/micropython-lib/blob/master/python-stdlib/html/manifest.py the package html has dependency string. Yet this relationship is reflected neither in the package index (https://micropython.org/pi/v2/index.json) nor in the html package metadata (https://micropython.org/pi/v2/package/6/html/3.3.3-2.json). Instead, the metadata indicates that string.mpy is part of the html package. The actual file seems to be shared with the string package (https://micropython.org/pi/v2/package/6/string/0.1.1.json)

What is the reason of this inlining? Isn't the package metadata file capable of expressing the dependencies? Does the html package get automatically updated when the string package gets new version or would it need manual approval?

[jimmo]

@aivarannamaa Sorry about the delay responding...

Happy to add the dependencies to the package index. I hadn't imagined any use for it there -- either you want the package or not, but maybe it's useful to know. Would be curious if you have any thoughts.

What is the reason of this inlining?

When you fetch a package metadata json (optionally at a specific version), then it's just telling you what you need.

Isn't the package metadata file capable of expressing the dependencies?

Sure, but then the client is just going to have to implement the recursive dependency walk. Might as well have the metadata just tell the client exactly what it's going to need up-front.

Does the html package get automatically updated when the string package gets new version or would it need manual approval?

Yes. When a package version is created (i.e. manifest.py is updated with a new version string), then the dependencies are "frozen" at their current version. In other words, once a versioned package json is created, it's never updated.

So if string is updated, the 'latest' html will get the new string version, but versioned html will keep their existing string dependency.

The underlying expectation here is that at a given point in time, every library in the repo is compatible with the current version of every other library.

[aivarannamaa]

Happy to add the dependencies to the package index. I hadn't imagined any use for it there -- either you want the package or not, but maybe it's useful to know. Would be curious if you have any thoughts.

I can imagine someone telling me I need html for a code example. I install it, spot html and string got installed. Later someone gives me another snippet, which requires string. Now I know that I already have string installed.

Another reason -- if I see that packages foo and bar both depend on baz, I'm more alert for possible conflicting expectations. When foo and bar both just inline (possibly different version of) baz, I may not even notice this. Disclaimer -- I haven't tried whether mip warns on overwriting an existing file.

but then the client is just going to have to implement the recursive dependency walk.

Isn't it already implemented in mip? And as the package format allows specifying dependencies, wouldn't another client need to implement it anyways?

When a package version is created (i.e. manifest.py is updated with a new version string), then the dependencies are "frozen" at their current version. In other words, once a versioned package json is created, it's never updated.

I suppose the same result can be achieved by freezing the dependency version in the metadata, instead of freezing the inlined module.

So if string is updated, the 'latest' html will get the new string version, but versioned html will keep their existing string dependency.

Just to be clear, by 'latest' html, do you mean installing without specifying the version or a 'newer' html? When the latest html version is currently 8.8.8, can it ever make a difference, whether I do mip.install("html") or mip.install("html", version="8.8.8")?

If a bug gets fixed in string, would you need to bump the version of html as well, if you wanted next mip.install("html")-s to benefit from it?