Adding Lockfile #728
Adding Lockfile #728
Conversation
|
Known limitations with the current implementation is as follows,
|
|
This seems like an interesting addition, but since we've linked this in a discussion with #727, I'll note that it is not an ergonomic solution for the problem that PR solves. They might be complementary, though! OT seeks the ability to dynamically swap implementations of virtual cores, so a set of targets expressed by in-tree fusesoc cores may have parts of the dependency tree swapped out for out-of-tree users. With the lockfile alone, we end up needing tooling to generate this file on the fly, something that is considerably more complex than simply adding required VLNVs on the command line (especially since this file covers much more than the virtual core mapping). |
|
I think lock files and providing the VLNV per command line serve two different purposes. From the OT perspective we are interested in solving the following scenario. An IP depends on a virtual core, i.e., the primitive library. We then wanto to to have an easy way (without core file change) to build this IP with different core providers. This can be for example the public |
|
Of course! I realize I didn't think through the use-cases properly. I do agree that it's complementary though. I would even suspect there are compelling use-cases for allowing a virtual/concrete mapping in the target sections for some use-cases as well. With that, I think the implementation in #727 makes sense but I'm thinking of some tweaks for consistency. Will write in more detail there. For this PR it would still be good to know if the lock file functionality matches what you are looking for. |
|
With the related discussion in #727 I suggest we do this in two steps. Let's focus this PR on reading and applying an existing (manually written) lockfile. For this, we also need the ability to specify the name of the lockfile to load. Once that is in place, we can figure out the remaining use cases for writing and updating lock files. |
|
I have changed the implementation to take a file path as lockfile argument, |
|
Could FuseSoC look for a lockfile in the 'cores root' directory unless overridden with a |
|
If no lockfile argument is supplied, FuseSoC could go through cores root directories and look for a specific file, |
|
Probably better to have a set place for it like most other package managers (e.g. Cargo, npm, uv, nix flake). They all have a top level config file (e.g. Cargo.toml) and in the same directory a lock file (e.g. Cargo.lock). In FuseSoC's case, the top-level config is Also for this to properly lock down dependencies we should really have hashes (normally sha256) for each core. |
There was a problem hiding this comment.
My first pass through the code. I haven't actually run the code.
I don't think we need the virtuals section. I think we only need a list of cores. A valid lock file will have an implementation for each virtual core dependency in it's list of cores.
I also don't think we should allow for partial lock files. I don't know of any package manager that allows them, and I don't really see the point of only partially locking down dependencies.
This looks closer to a requirements/dependencies format (e.g. requirements.txt) than a lockfile. Most language package managers seem to agree on a way to do this, touched upon in #728 (comment). They express requirements in their config (e.g. pyproject.toml) then lock this down with an exact version of each library/core accompanied by a hash, source location and dependencies in a lock file (e.g. uv.lock). I think this way of managing dependencies would work well for FuseSoC.
|
@HU90m All valid concerns. I'll add my comments to each of the topics. hashes: We want hashes for several reasons. Signing of cores is the main thing I'm thinking about which is becoming more relevant once we have a larger body of publicly available cores, but also detect modified caches etc. But we don't have support for that yet, and I think we should aim to have a first version of lock files without that and extend with hashes and source file locations later. partial lock-files: I personally see great value in this in situations where you need a specific version of a core instead of the one that FuseSoC would pick by default, e.g. to temporary work around tool bugs. It's good that you bring up requirements.txt, because I believe my vision is probably somewhere between that and cargo/poetry lock files. Lock file location: The FuseSoC use model is quite different from the sw package managers in this regard. virtuals in lock files: You said we don't need the virtuals section at all because the cores section will tell us which concrete implementation to use. I think that could work under the following assumptions. One problem however arises when you have two concrete implementations implementing the same virtual because you can't explicitly tell which one to use. And I do think that's a valid use-case. That was the idea of the two sections, |
tests/test_coremanager.py
Outdated
| @@ -349,7 +349,6 @@ def test_virtual_non_deterministic_virtual(caplog): | |||
| core_manager=cm, | |||
| work_root=work_root, | |||
| ) | |||
| edalizer.run() | |||
There was a problem hiding this comment.
Is this an intentional change, and if so, is it relevant to the other code or should it be in a separate commit?
There was a problem hiding this comment.
It is intentional, but not relevant to the other changes. I will leave it out.
- Removed lockfile_store - Removed invalidation of lock file
Okie, happy for hashes to be added in a later.
Can you not just specify the specific version of the core you want to depend on in the core file. or are you wanting to override a core file's choice?
I don't think the lock file should be for manually setting mappings. Or if it is it shouldn't be called a lock file to avoid confusion with other tools lock files which are only ever meant to be generated. |
Current FuseSoC OntologyAdded this section so you can check my understanding of FuseSoC at the moment.
Specifying core versions and virtual core mappings
Adding a lock filePersonally, I would like a lock file to function on the library level to manage external dependencies like in other package managers. It could be associated with a Prior ArtCargo.toml# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "aho-corasick"
version = "1.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916"
dependencies = [
"memchr",
]
[[package]]
name = "cc"
version = "1.1.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72db2f7947ecee9b03b510377e8bb9077afa27176fdbff55c51027e976fdcc48"
dependencies = [
"shlex",
]uv.lockversion = 1
requires-python = ">=3.6, <4"
[[package]]
name = "attrs"
version = "22.2.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/21/31/3f468da74c7de4fcf9b25591e682856389b3400b4b62f201e65f15ea3e07/attrs-22.2.0.tar.gz", hash = "sha256:c9227bfc2f01993c03f68db37d1d15c9690188323c067c641f1a35ca58185f99", size = 215900 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/fb/6e/6f83bf616d2becdf333a1640f1d463fef3150e2e926b7010cb0f81c95e88/attrs-22.2.0-py3-none-any.whl", hash = "sha256:29e95c7f6778868dbd49170f98f8818f78f3dc5e0e37c0b1f474e3561b240836", size = 60018 },
]
[[package]]
name = "click"
version = "8.0.4"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "colorama", marker = "platform_system == 'Windows'" },
{ name = "importlib-metadata", marker = "python_full_version < '3.8'" },
]
sdist = { url = "https://files.pythonhosted.org/packages/dd/cf/706c1ad49ab26abed0b77a2f867984c1341ed7387b8030a6aa914e2942a0/click-8.0.4.tar.gz", hash = "sha256:8458d7b1287c5fb128c90e23381cf99dcde74beaf6c7ff6384ce84d6fe090adb", size = 329520 }
wheels = [
{ url = "https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl", hash = "sha256:6a7a62563bbfabfda3a38f3023a1db4a35978c0abd76f6c9605ecd6554d6d9b1", size = 97486 },
]package-lock.json{
"name": "gubbins-compiler",
"version": "0.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "gubbins-compiler",
"version": "0.0.0",
"dependencies": {
"@djot/djot": "^0.3.2",
"happy-dom": "^16.5.3",
"highlight.js": "^11.11.1",
"mime": "^4.0.6",
"temml": "^0.10.32",
"ws": "^8.18.0"
}
},
"node_modules/@djot/djot": {
"version": "0.3.2",
"resolved": "https://registry.npmjs.org/@djot/djot/-/djot-0.3.2.tgz",
"integrity": "sha512-joMKR24B8rxueyFiJbpZAqEiypjvOyzTxzkhyr0q5mM/sUBaOD3unna/9IxtOotFugViyYlkIRaiXg3xM//zxg==",
"license": "MIT",
"bin": {
"djot": "lib/cli.js"
},
"engines": {
"node": ">=17.0.0"
}
},
"node_modules/happy-dom": {
"version": "16.6.0",
"resolved": "https://registry.npmjs.org/happy-dom/-/happy-dom-16.6.0.tgz",
"integrity": "sha512-Zz5S9sog8a3p8XYZbO+eI1QMOAvCNnIoyrH8A8MLX+X2mJrzADTy+kdETmc4q+uD9AGAvQYGn96qBAn2RAciKw==",
"license": "MIT",
"dependencies": {
"webidl-conversions": "^7.0.0",
"whatwg-mimetype": "^3.0.0"
},
"engines": {
"node": ">=18.0.0"
}
}
}
}A fusesoc-y lock formatThe FuseSoC lock file could look something like this (after all the features are added):
version: 1
libraries:
- name: "fuseoc-cores"
location: "."
url: "git+https://github.com/fusesoc/fusesoc-cores"
rev: "cb967637dbf7cee25117203bbdf9c10b62dfb25a"
hash: "sha256-0JLnk+qr991aKWjzTrG4JE0L5e2NrcOstT8+ggjzKr0="
cores:
- "::serv:1.0.0"
- "::serv:1.0.2"
- name: "opentitan-ip"
location: "hw/ip/"
url: "git+ssh://github.com/lowRISC/opentitan"
rev: "f7ce10b62dfb2e2511cb967637db7203bbdf9c5a"
hash: "sha256-1a0L5e2NrcOsKWjzTrG4JEtT8+ggjzKr00JLnk+qr99="
cores:
- "lowrisc:ip:aes:1.0"
- name: "opentitan-prim-generic"
location: "hw/prim/generic"
url: "git+ssh://github.com/lowRISC/opentitan"
rev: "f7ce10b62dfb2e2511cb967637db7203bbdf9c5a"
hash: "sha256-1a0L5e2NrcOsKWjzTrG4JEtT8+ggjzKr00JLnk+qr99="
cores:
- "lowrisc:prim:sram"
providers:
- core: "::serv:1.0.0"
url: "git+ssh://github.com/olofk/serve"
rev: "b2e2511cb967637db7203bbdf9c5af7ce10b62df"
hash: "sha256-KWjzTrG4J0JLnk+qr9EtT8+ggjzKr091a0L5e2NrcOs="
# Not required if the dependency resolution is always consistent,
# but may be nice to include.
cores:
- name: "lowrisc:ip:aes:1.0"
location: "hw/ip/aes/aes.core"
dependencies:
- "lowrisc:ip:tlul:0.1"
- "lowrisc:ip:lc_ctrl_pkg:0.1"
- "lowrisc:ip:edn_pkg:0.1"
- "lowrisc:ip:keymgr_pkg:0.1"
- name: "lowrisc:ip:tlul:0.1"
location: "hw/ip/tlul/tlul.core"
dependencies:
- "lowrisc:tlul:socket_1n:0.1"
- "lowrisc:tlul:socket_m1:0.1"
- "lowrisc:tlul:adapter_sram:0.1"Associated with a [library.fusesoc-cores]
sync-uri = https://github.com/fusesoc/fusesoc-cores
sync-type = git
rev = "cb967637dbf7"
[library.opentitan-ip]
sync-uri = https://github.com/lowRISC/opentitan
sync-type = git
location = "hw/ip/"
rev = "Earlgrey-PROD.M6"
[library.opentitan-prim-generic]
sync-uri = https://github.com/lowRISC/opentitan
sync-type = git
location = "hw/prim/generic"
rev = "Earlgrey-PROD.M6"
[library.a-local-library]
location = "rtl/"A TOML version (not proposing, but adding for those more toml literate).version = 1
[[library]]
order = 0
name = "fusesoc-cores"
location = "."
url = "git+https://github.com/fusesoc/fusesoc-cores"
rev = "cb967637dbf7cee25117203bbdf9c10b62dfb25a"
hash = "sha256-0JLnk+qr991aKWjzTrG4JE0L5e2NrcOstT8+ggjzKr0="
cores = [
"::serv:1.0.0",
"::serv:1.0.2",
# ...
]
[[library]]
order = 1
name = "opentitan-ip"
location = "hw/ip/"
url = "git+ssh://github.com/lowRISC/opentitan"
rev = "f7ce10b62dfb2e2511cb967637db7203bbdf9c5a"
hash = "sha256-1a0L5e2NrcOsKWjzTrG4JEtT8+ggjzKr00JLnk+qr99="
cores = [
"lowrisc:ip:aes:1.0",
# ...
]
[[library]]
order = 2
name = "opentitan-prim-generic"
location = "hw/prim/generic"
url = "git+ssh://github.com/lowRISC/opentitan"
rev = "f7ce10b62dfb2e2511cb967637db7203bbdf9c5a"
hash = "sha256-1a0L5e2NrcOsKWjzTrG4JEtT8+ggjzKr00JLnk+qr99="
cores = [
# ...
]
[[provider]]
core = "::serv:1.0.0"
url = "git+ssh://github.com/olofk/serve"
rev = "b2e2511cb967637db7203bbdf9c5af7ce10b62df"
hash = "sha256-KWjzTrG4J0JLnk+qr9EtT8+ggjzKr091a0L5e2NrcOs="
# Not required if the dependency resolution is always consistent,
# but may be nice to include.
[[core]]
name = "lowrisc:ip:aes:1.0"
location = "hw/ip/aes/aes.core"
dependencies = [
"lowrisc:ip:tlul:0.1",
"lowrisc:ip:lc_ctrl_pkg:0.1",
"lowrisc:ip:edn_pkg:0.1",
"lowrisc:ip:keymgr_pkg:0.1",
# ...
]
[[core]]
name = "lowrisc:ip:tlul:0.1"
location = "hw/ip/tlul/tlul.core"
dependencies = [
"lowrisc:tlul:socket_1n:0.1",
"lowrisc:tlul:socket_m1:0.1",
"lowrisc:tlul:adapter_sram:0.1",
# ...
] |
|
Your ontology is mostly correct @HU90m . Just a few comments before we move on
While it's true that the dependencies are defined per fileset, I would just say a fileset is a collection of files. The ones from other cores are other filesets. Dependency resolution happens on the core level
Technically, version is set to
That's really a detail of the sat solver in the dependency manager, but I would say in that in practice it will select the highest version that fulfills the constraints of the dependency tree.
This has been supported since 2016 :) I have given a lot of thoughts to the idea of library-level lock files. I do think there is some merit to this, but I think there are cases when this is hard to enforce. Imagine e.g. when we have a public library á la pypi. It is probably problematic to go back in time in that case. Or local cores that don't have a version-controlled backing. The good news however is that you already have more or less everything you need. The library sections in fusesoc.conf has a key called When it comes to virtuals, I still think we should have the possibility to specify these per target (e.g. for when you want to set tech-specific primitives for a specific FPGA), on the command-line (for quick overrides or when decided by an overarching build flow) and with a file that can be loaded from the command-line (for when you have a lot of different virtuals and want to switch between them easily. I think we're mostly there but I want to pull in rudimentary lock file support first while we think through the last use-cases for virtual mapping. So with that said, we're starting with adding support for loading core-level lock files and from that we can add more features (hashes etc), add support for writing lock files and nail down the virtuals syntax. |
- Remove unused import - Removed isinstance check for _lockfile - Removed virtual core mapping of cores in lock file
|
Thanks for considering my ideas. Let's get the cores based lockfiles polished up and merged. Could I suggest we make the cores list a list of maps/dictionaries, e.g. lockfile_version: 1
fusesoc_version: 2.4.2
cores:
- name: ":lib:pin:0.1"
- name: ":lib:gpio:0.1"
- name: ":common:gpio_ctrl:0.1"
- name: ":product:toppy:0.1"To make it easier to include more information in the future, such as hashes cores:
- name: ":lib:pin:0.1"
hash: "sha256-1a0L5e2NrcOsKWjzTrG4JEtT8+ggjzKr00JLnk+qr99=" |
tests/lockfiles/works.lock
Outdated
There was a problem hiding this comment.
Could we make the idiomatic file extension .lock.yml so it's easier for editors to know that it's a YAML file?
I can map .core to YAML in my editor, but .lock is a little to common to do this for.
- Test case cleanup
Yes, let's go with that. As you say, we know already now that there will be more fields coming. This also reminded me that we need to decide how to treat multiple entries trying to lock the same core. Given that it's a dict, the order is technically undefined so we can't rely on any ordering. I suggest making it an error if there are multiple entries locking the same thing. Fine with everyone? |
|
The suggested change is a list of dicts. We could make it a dict: lockfile_version: 1
fusesoc_version: 2.4.2
cores:
":lib:pin:0.1":
":lib:gpio:0.1":
":common:gpio_ctrl:0.1":
":product:toppy:0.1":which becomes the following in python: {
'lockfile_version': 1,
'fusesoc_version': 2.4.2,
'cores': {
":lib:pin:0.1": None,
":lib:gpio:0.1": None,
":common:gpio_ctrl:0.1": None,
":product:toppy:0.1": None,
},
}This'll mean the yaml parser can handle duplicates adhering to the yaml standard, and there'll be less logic in FuseSoC. Note other config languages, such as TOML, don't have a unit type (null/None). When there are fields it would look like this: cores:
":lib:pin:0.1":
hash: "sha256-1a0L5e2NrcOsKWjzTrG4JEtT8+ggjzKr00JLnk+qr99="
":lib:gpio:0.1":
hash: "sha256-1a0L4e2NrcOsKWjzTrG4JEtT8+ggjzKr00JLnk+qr99="I don't have a strong opinion either way.
|
|
I think we stay with a list of dicts. Not a big deal, but slightly nicer syntax when we know we will need other options as well. Agree on the point that we get duplicate key checking for free with the dicts instead, but I think we might want to extend this to work with multiple lock files later on (e.g. one for common dependencies and different for synth/sim targets) and in that case we need to handle duplicate keys anyway. |
|
Noticed some unhandled exceptions when using malformed lock files. Let's fix those, but other than that I think we're getting ready to pull this in. |
|
I'll get to work on the mapping file now, so we can remove the partial lock file feature from this PR |
FuseSoC Lockfile
Format
The format contains a list of locked down versions for the cores in the design. There is also a map describing mapping of virtual cores in the design.
Schema
Following is the JSON schema for the lock file format.
{ "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "FuseSoC Lockfile", "description": "FuseSoC Lockfile", "type": "object", "properties": { "cores": { "description": "Cores used in the build", "type": "array", "items": { "type": "string" } }, "virtuals": { "description": "Virtual cores used in the build", "type": "object", "patternProperties": { "^.+$": { "description": "Virtual core used in the build", "patternProperties": { "^.+$": { "description": "Core implementing virtual core interface" } } } } }, "fusesoc_version": { "description": "FuseSoC version which generated the lockfile", "type": "string" }, "lockfile_version": { "description": "Lockfile version", "type": "integer" } } }File location
The file is located at the current working directory of the
fusesocrun.Command line argument
An command line argument has been added to
fusesoc run,lockfile. Thelockfileargument can hawe following values wereenableis the default.--lockfile="enable"--lockfile="disable"--lockfile="reset"Enable
Enables the use of lock file. If there is no lock file to be found one will be generated during the
fusesocrun. If there is a lock file, the file will be read and used to influence the core selection during the run.Core version selection
If there are dependencies which relation is not locked down to a single version, the use of lock file will change that relation to the one in the lock file. If the range of versions described by the dependecy is outside that described in the lock file, the lock file version will not be used and the lock file will be marked as invalid.
Virtual core selection
The virual core mapping in the lock file will be applied to virtual core dependecies during core selection.
Invalidation
During the fusesoc run, the validity of the lock file is evaluated. If invalid versions, missing cores are detected the lock file is invalidated. When invalidated the lock file is re-written at the end of the run.
Disable
The use of lock file will be disabled, no loading or storing of lock file.
Reset
No lock file is loaded. A new lock file is stored after core selection.
Use cases
Locking down a core dependency
The user might want to lock down a dependency, by writing a simple lock file with the core that shall be locked down.
Shared development
Multiple developers developing a design, might want to share lock file to limit any dependecy ambiguities.