Manage modules and themes with composer in addons/.

[Daniel-KM]

When using docker containers, the question to store the modules and the themes in a volume or inside the container is always complex, some projects prefer one way and some other the other way. So the possibility to use a specific directory /modules/custom/ and /themes/custom/ is a good solution to many uses cases. Some other applications choose /modules-custom/ and /themes-custom/ or some other alternatives, but the first seems simpler to understand. This is the solution used by drupal.
Furthermore, this is the first pr to finalize using one single composer for all omeka (composer used to manage modules and themes natively), while keeping old modules compatible.

[zerocrates]

So the idea here is that this allows you to have some modules or themes in the container but also mount a volume and easily have others. Am I understanding that right?

I would imagine this will require some work in the AssetUrl helper at least, which makes assumptions about what paths to files inside plugin modules/themes look like. AssetUrl already looks up the actual module and theme objects to get versions so getting their real base path shouldn't be a big deal.

I'm not 100% convinced/certain of the actual path though... I feel like keeping the actual modules and themes folders so all their contents are expected to be modules and themes might be best. I could see maybe custom/themes and custom/modules maybe?

The connection with composer, I imagine you're thinking to have composer install the modules/themes included in the composer.json into either the "custom" or "regular" path. Which are you thinking? Maybe informs what a good name would be.

[Daniel-KM]

In most of my project, i always include major modules (value suggest, custom vocab, easy admin, etc.) and i have one or two custom modules (even if i try now to include all new features in existing modules), and a custom theme, lightly or from scratch. So, yes, the aim is to include common modules inside the container (so upgrade as a whole) and a volume for specific modules and the same for themes. It's important for themes, because users want generally to modify it (even if css editor helps).

For the structure, the choice of drupal is three modules/ modules/custom/ modules/contrib/ + hierarchical for sites. For wordpress there are no custom dir. For jommla, there are components, modules and plugins. Symfony and laravel enforce composer, included custom packages. So i proposed to have main modules in /modules/ (the ones that are used anywhere) and the other ones in /modules/custom/, but indeed, it may be /modules_custom/ or custom/modules/ as you say, simpler because all custom are in a specific dir, so one volume to manage.

In fact, maybe this pr is too early, and it may be simpler to use composer for all modules (vendor/xxx/yyy/ (mixed)), or vendor/xxx/yyy and vendor/modules/xxx/yyy), so in that cases modules/ will be custom or old modules, but it may hard to override a vendor module.

And it may be the same for themes, that are often customized. Inheritance of themes may be useful, but slower. So something to think about.

So the aim is to think about that before pr in https://github.com/composer/installers, where you can find a lot of other types and directories for many apps.

[zerocrates]

Not planning to work on this right away but in the way of notes: I'm not at all set in stone on this but to my current thinking this kind of parallel folder setup could make sense, but I think my preference on loading/names would go something like, the existing modules and themes folders stay for the current method of loading and installation, and have it be Composer-installed ones that use a new path... composer-addons/modules, composer-addons/themes, or something like that.

That feels like it's more likely to offer a smooth upgrade path. Of course, there's more to work out. A separate folder isn't required for composer to work, but might be desirable on its own, particularly for the container-related reasons mentioned in previous comments. There'll also be more process/convention-type stuff to think of: there are options for doing things like merging multiple Composer files, which might be desirable to keep a particular install's list of required addons separate from the core's own requirements; will modules continue to support both installation styles or require installation through Composer, things like that.

I'll note here also that we do have a Composer plugin that ships with Omeka S for installing addons: right now it installs to the "normal" folders as that's all there is. Changing where that existing plugin puts things is a good first step for those trying things out in this area.

[Daniel-KM]

I fully implemented automatic install with composer in the directories addons/modules/ and addons/themes/ and manual install (via unzip or git) in modules/ and themes/, that takes priority when there is duplicate.

So now easy to use modules/ and themes/ as volume for a container. And a lot easier to manage Omeka.

And side benefits: performance is better during init, since data about modules are stored by composer one time, so no need to load all composer.json and all module.ini and to analyse them all for each request.

Only the first commits were needed for the feature, but in practice, there was issues with the overriding mechanism, the module Common, the asset dependencies, and the possibility to continue to manage modules and themes without composer, so there are a lot of commits next to implement/fix these points. There are a lot of tests too (built automatically by my ide), because it is an important core feature.

Note: this pull request is fully compatible with current omeka behavior (it was a hard point), so it can be included in 4.2.1, without waiting for 4.3. So the upgrade is very smooth. Another important feature that can be included in 4.2.1 without any impact is the one about roles in config and the one about full jsonSerializing() of representations (only rare modules should be fixed and i can push a pr for them), but it is another subject.

I already published the module Common on packagist (https://packagist.org/packages/daniel-km/omeka-s-module-common), because it was a module that caused issues, that are fixed now. Future versions of modules won't have to "require" it, only "use" it, as any module. So module can have dependency: for example, once updated, the module Urify will install the module Value Suggest automatically.

Note that on packagist, omeka/omeka-s is already published, but it was declared with type "project" (in composer.json). So it explains the need to use "omeka/omeka-s-core" in modules. I added a readme in addons/readme.md.

Last point: i prepared the commit for composer-installers, that has no very original code and that retakes the same code than the internal composer install plugin.

So i wait for your confirmation or your comments to publish new modules and composer installer. There are Test Override and Test Override theme too, but only to test the current feature.

Note: This feature was funded for the digital library Manioc (back end is omeka s, front end is drupal) of the Université des Antilles (subvention Agence bibliographique de l’enseignement supérieur Abes) and they ask me to indicate it.

[zerocrates]

It'll be a little bit before I can fully review this. A couple immediate pieces of feedback from my quick look:

• I do think we'd like composer-addons as the folder where Composer will put stuff, just for maximum clarity
• Your bit about omeka/omeka-s being a "project" type and therefore needing a provides and all that... we don't experience that as a problem right now: the default theme specifies omeka/omeka-s as a dependency with no problem, and Composer installs it with no complaint. What error or problem do you get when doing that?
• The "omeka-assets" composer plugin seems out of scope to me and I don't think we'd want to include it in the core
• For addons that need to have a different installed name because their repo name isn't simply the same as their folder name, I don't love the code that tries to guess or inflect the name, in general I like to avoid that kind of inflection. Instead I think I'd simply have "custom-named" packages use the existing install-name extra key to explicitly set their name (and of course its the composer package name that's relevant here so it's also possible for people to just pick a vendor namespace that results in the addon name being the package name and avoids the issue)
• I don't feel great about having the core have specific support for handling Common, but I'll think about this... obviously it does improve compatibility to have it. I could imagine maybe this support plus the "assets" one being one or more plugins that you provide as your own package(s) to get the same support while keeping the core from having code specific to one module.
• I think maintainability is going to demand some sort of separation between the installation-specific stuff like modules and themes and omeka-s's own dependencies. Probably the most straightforward way to accomplish this will be to use wikimedia/composer-merge-plugin, so installation-specific requires can live in their own file. I can think of some other options, like having the addons be their own separate composer project, or having one from the "outside" that requires both omeka-s and the modules/themes... but those both feel like they have significant downsides.
• I'm not 100% convinced on reading the INI data from the Composer data, even optionally. I might leave all that out, at least initially.
• I'll check on this but I'll bet that we'll need to ship a composer.json for the core with a version entry: Composer can guess the current version from Git, but lots of people aren't installing with Git. I'll have to think on whether I want to maintain that or have the tooling that makes the core zips put a version entry in there. Of course if everything just doesn't bother requiring omeka/omeka-s at all, or uses a * version specifier this isn't an issue, but that seems kind of contrary to the whole exercise here.
• The whole question of overriding between "same" modules in both the Composer and non-Composer locations is a little fraught... even if you override the module, its dependencies are going to get loaded from the regular Composer autoloader... this feels like it's courting trouble down the line but I'm not totally sure whether it's worth worrying about.
• I'm not sure if "standalone" makes sense to cater to here... I can see certainly how modules could be happy with their current standalone nature, but then I'd think they'd just keep distributing as they do now? In general I imagine to have them work either way should be a pretty minor adaptation for the module... for example loading their local autoload only if it exists. And if they're just hard-requiring that they're "standalone" installed then I wouldn't expect them to be listed on Packagist.

Maybe it's clear already, but I'm going to be pretty picky on this one since the dependency stuff generally and the addons are such a key component.

More as a side note: we have control now of our "omeka" namespace in Packagist (which for boring reasons we didn't before) so it's more feasible to extract things like the installer composer plugin we have now into an actual separate package.

[Daniel-KM]

Thanks for the response. So some quick remarks too:

• Ok for the directory "composer-addons", i prefer short one, but it doesn't matter a lot.
• Ok, if you can create a composer plugin "omeka/composer-addon-installer" outside of omeka, it may be simpler for future, in particular to include features that you don't want natively. For "provide"/"project", i'm going to reexamine it.
• Ok, i'm going to extract the omeka-assets feature as a composer plugin daniel-km/composer-plugin-omeka-assets (or maybe a short name), that will be used by modules that needs it.
• Ok for standalone, i'm going to remove it (or to manage it via another composer plugin).
• For the inflexion in the name, it is very common in composer-installers, so i adapted the MediaWiki one, but you can find a lot (cakephp, cockpit, dokuwiki, grav, october, ontowiki, etc.). It's not common to create multiple repositories for an organization, so the use of prefix/suffix helps a lot to organize repositories. But if you really prefer, i can remove it too. Note that the official key for that when there is no inflexion is installer-name, not install-name as in the current omeka plugin (but i added it for compatibility).
• For the inclusion of Common, it is to fix the php require_once i added in other modules (the methods are needed to simplify install before the module is managed). The last version of common (3.4.77) already includes improvement to make the inclusion useless, but for smooth upgrade, it is better to check it early, so it avoids issues during transition (but once the feature, i'll publish an update of all my modules, but people may mix them). Or maybe a small patch to manage the point differently somewhere else in omeka may be enough.
• For separation of packages and the mediawiki composer-merge-plugin, i have to check.
• for ini : the pr let the choice and pure ini is still supported. The use of composer avoids to check version and constraint on every request and every module. The other point is it is useless to copy the same data in composer.json and module.ini, it is a source of confusion, and there is already a full file that contains the same (installed.json). Anyway, i'll let the module.ini and the composer.json in my modules for some time. And finally, there is no more extra key, except "installer-name" (an official key) and "label" (not existing in composer and the only one needed to replace fully module.ini) and "configurable" (that can be skipped or determined automatically by omeka). The ones for theme are less important, since only the current theme is read.
• when there is no git or no version, the version is set to 1.0.0, according to composer. Composer recommends to not set a version, but not forbid it, so there is no issue.
• for the management of overriding, it is managed magically natively via a small piece of code in bootstrap.php. And it works fine even with complex tests. The use case is that i have a container with main modules and i let user install their own modules in modules/, so i know that they will install duplicates or new versions. But something to think about more if needed.
• the key "standalone" can be removed, it was designed for complex install, but finally the override mechanism works fine. And it can be a specific composer.json when needed.

The feature is done in a way that it does not change anything for current installations and modules for smooth evolution.

[zerocrates]

• I'll think it over on the inflection feature.
• If "installer-name" is the key used by the main installers plugin then I don't mind standardizing on that, at least to be allowed, to smooth the path if we do shift over to composer-installers.
• The INI vs. composer.json, I'm not necessarily against it. Obviously things like the required Omeka version and much of the rest of what's in the INIs have straightforward equivalents in composer.json, but I think I'd rather see it separated into its own feature. Not to say that it couldn't still be there even in a first release with this support, but I'd like to focus on what's necessary first. Changes here also mean us changing our tooling for stuff like the addon listings on the website and we'd have to be sure we're prepared for that stuff to update alongside any change here, even an optional one.
• On Common, could Common itself require a composer plugin that sets up your symlink? Maybe that's the cleanest option. Having the core have knowledge of specific modules is really not a road we've gone down at all even for our own and it's one I'd like to stay away from. I think there's an issue doing that not in the root package though... we could maybe put the theoretical plugin here into allow-plugins but not ship it. Of course it's not a problem at all if Composer just installs to the existing modules and themes folders, which it can definitely handle being an intermix of composer-managed and non (this is already the case for the default theme). I don't have a strong handle on how important or not that separation is to people's intended container workflows, etc.
• Yeah, I've seen the documentation on composer defaulting to 1.0.0 for the root package. It's just something we'll need to account for one way or another. We already set a version number manually in the code so just shipping with an explicit one isn't so onerous. I'll think about what makes the most sense. My guess is that even in the Git checkout case, if you're just on develop you'll run into problems with version matching as composer will think the root version is "dev-develop" and not match "^4.2.0". branch-alias is another solution here but just setting the version may be simplest, as we'll want it for non-checked-out installs.

[Daniel-KM]

I removed most of the code (extra keys, common, composer plugins, etc) you what don't want.
I replaced "provide" by "branch-alias", but it may be removed too, since there is no dev, no roadmap, and a full release can follow an alpha version.
For the inflexion, it is easier to identify package by people, but if it is automatic, it is less important.
For the omeka-assets, i published https://gitlab.com/Sempia/composer-plugin-external-assets (require sempia/external-assets).
I moved too the key "configurable" from module.ini to module.config.php, that is more logical (and may be a separate pr).

So remaining in the pr :

• the main feature to separate modules/ and composer-addons/
• overriding mechanism for duplicate add-ons

The reflexions that remain:

• use, or not, composer-merge-plugin (so two separate composer.json, a little slower) ; drupal used it, but replaced it by composer path repositories (one composer for multiple paths without install plugin); drupal uses site by core as a dependency of the user project too, like laravel and symfony, but too much complex for now. Or any other solution of course.
• the fact that module.ini and composer.json have many duplicate keys, but it doesn't matter a lot.

[Daniel-KM]

And for inflexion : in fact, the main name in composer.json can be anything , with any vendor and project name, even if they don't exist and it is not related to the repository, but it is simpler to use the same anywhere and for search on packagist (it lists all omeka-s-module-xxx together). So the choice is between automatic inflexion or manual installer-name.