Reason for preference of .py over .mpy

[peterhinch]

Given a project where foo.py is too big to compile on the target you cross-compile to foo.mpy and copy that to the target. Alas the problem remains until you rename or move foo.py.

Is there a reason why MP builds a .py file when a .mpy is available?

[jimmo]

Is there a reason why MP builds a .py file when a .mpy is available?

@peterhinch assuming you mean "loads" here (not "builds").

Good question! This definitely seems to cause some confusion, and I think loading the .mpy preferentially does make more sense.

Nervous that this would be a backwards incompatible change though... maybe something we can change for MicroPython 2.0?

[bulletmark]

Wouldn't it make most sense to load the latest file?

[mattytrentini]

On-device filesystems often don't support timestamps (to reduce flash consumption).

[dhalbert]

If I have a precompiled .mpy library, but want to test changes to it or add debugging code, I find it easier to leave the .mpy there and add a .py which I can edit, and then remove later. This is how we explain the priority of .py over .mpy to CircuitPython users. I think the confusion about the priority exists either way.

[peterhinch]

To explain why this is a nuisance here is my situation. I have a project mico-gui with multiple files. The core file ugi.py will compile on many platforms but if the difference between the host RAM size and the size of the frame buffer is too small, it will not. The file is referenced by many modules. The following would need to be implemented in every widget and every demo:

try:
    from ugui_cc import stuff  # .mpy
except ImportError:
    from ugui import stuff  # .py

Failing that, my user manual needs to say that if ugui.py does not compile on the given hardware, it should be renamed and cross-compiled to ugui.mpy. While this works, it means that the user's source tree is modified and a git pull will bring in a new ugui.py. An unwary user will get compile failures.

While the workround is hardly rocket surgery it is inelegant. It would be good if there were a way to tell MP to prefer loading .mpy files.

[dhalbert]

I see that you do not make tagged releases for your libraries. For CircuitPython libraries, we tag (with a semver) a release on GitHub of each library whenever we make a change. There are release artifacts that are .zip's of the .mpy's for the library (no .py in the artifact), mpy-crossed and labelled for each version of CircuitPython that is currently supported. A .py zip is also generated. Example.

We have found that tagged releases are vital when doing support. We can ask someone if they are using the latest, recommend a rollback, etc. We recommend people use the .mpy versions unless they want to do modifications. There is a __version__ inside the library to label its version.

The .mpy libraries are also all collected in a "bundle" which is all the libraries in a category: Adafruit libraries, or community-supported libraries. Bundles are produced nightly as necessary. Python-source bundles are also available. The circup tool (similar to upip, etc., I think) will load specified libraries from the latest bundle. Circup can notice if a release is out of date and update accordingly.

https://circuitpython.org/libraries has a summary of all this and links to the bundles.

[peterhinch]

Thank you. Food for thought.