Modular services

[roberth]

I invite you to collaborate on modular services. (EDIT: and you can make PRs or push to this branch)

A lot is described in nixos/doc/manual/development/modular-services.md (see PR files), but in a nutshell:

• Adds a system.services tree where arbitrary services can be added by means of imports
• Is inherently multi-instance
• Uses modules not just for placing option trees, but for users to import
• Allows individual services or clusters of sub-services to be configured
• Provides a possible basis for portability and connections to be added
  • perhaps https://discourse.nixos.org/t/pre-rfc-decouple-services-using-structured-typing/58257

As an example, I've ported one service, ghostunnel to have a modular service.
Some services will be more complicated, requiring more than the current facilities. These can be added.

This PR does not require that all services become modular services.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[aanderse]

Some services will be more complicated, requiring more than the current facilities. These can be added.

let's elaborate on that so people who may not be as familiar with the history of this PR up to this point stand a chance at understanding this

the new system.services module in this PR currently does not currently have, but could require the ability to:

• create system users
• place files under /etc
• set runtime parameters of the linux kernel, as set by sysctl
• add packages to the global system environment
• open ports in the system firewall

the above is just a list of the most common/useful patterns that many nixos service oriented modules use, but anyone else should feel free to edit this comment and add to it if they feel it useful to do so - as well as check off the functionality if they contribute it to this PR

---

thanks for getting the ball rolling on this one @roberth 🙇‍♂️

[ibizaman]

I really like the idea and the way services can be imported in the top-level attrsOf. I'm already looking forward to having multiple instances of a database!

From a theoretical point of view, I understand the nesting of services but I'm not sure to understand the use in practice. Would this be used to provide opinionated coupling between services? For example there would be a service grouping Nextcloud and Postgres and others to provide an out-of-the-box experience? And this would allow to split that big module into multiple ones?

You mentioned the pre-RFC on decoupling services and I'm the one that wrote it. There are some common goals here and I'd love to collaborate. One way would be to write the systemd implementation of the generic service using the structural typing I propose in the pre-RFC. But there are still questions to be elucidated there so not (yet) sure it's the ideal solution. And also, I don't want to pull the rug on "my" side for no good reason. I know you're way more proficient in the module system than I am so I'd love to get your input on how we could work together (us - the community) so that both RFCs can cooperate, merge? At least not hinder one another.

I had also a few noob questions about modules. I've been using modules a lot but not that deep.

Thanks for progressing on this!

[ibizaman]

+1 to keep the distinction. I had cases where I wanted one or the other.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-decouple-services-using-structured-typing/58257/16

[roberth]

I understand the nesting of services but I'm not sure to understand the use in practice. Would this be used to provide opinionated coupling between services? For example there would be a service grouping Nextcloud and Postgres and others to provide an out-of-the-box experience? And this would allow to split that big module into multiple ones?

This can be used for groupings of multiple "off-the-shelf" services into a common configuration, or for non-trivial services that consist of multiple executables, such as apache cassandra, or even a mix, such as pulsar.

I think the distinguishing factor is that when a sub-service is an off the shelf one, such as a database, admins may prefer to have a single db service with multiple apps that consume it. @ibizaman this is where I think the ideas you've worked out apply.
This could be a configurable thing as well - a boolean in the "parent" service that controls whether sub-services are generated to fill the requests.

I'd love to get your input

Of course! I will have a look soon-ish.

a few [...] questions

Happy to help :)

[ibizaman]

I think the distinguishing factor is that when a sub-service is an off the shelf one, such as a database, admins may prefer to have a single db service with multiple apps that consume it.

Agreed. The end user should have the choice of either an instance per service or one instance for all of them.

This can be used for groupings of multiple "off-the-shelf" services into a common configuration, or for non-trivial services that consist of multiple executables, such as apache cassandra, or even a mix, such as pulsar.

For non-trivial packages with multiple binaries, I indeed don't see how contracts can apply. But if we're talking about multiple services working together, this is where our work could be merged and become even more powerful. If one write a group of services, like you allow, and the link between those services are contracts, then we can easily swap one of those services for another one.

[ElvishJerricco]

system.services is probably too close to systemd.services. People already get confused by toplevel services and systemd.services. Not sure what would be better though :/

[roberth]

Perhaps systemServices is sufficiently distinct?

Adding any extra words is probably a net negative.

Explaining extra words

System

It's the place where service modules are connected to the system (as in e.g. systemd --system).
Perhaps this term could be replaced, but I can't think of a synonym for "system services" that is nearly as good.
Host comes close, but that gets weird with VMs or "hosting a configuration" in the sense of it being deployed onto something.

Module

Putting the word module or modular in there would be wrong, because that option is where the modules are consumed and turned into a concrete configuration, such that the modules become irrelevant.
For example, this would be valid naming:

{ serviceModules, ... }:
{
  systemServices.etcd = { imports = [ serviceModules.etcd ]; };
}

Portable

Portability is similarly irrelevant, because it is a property of a module, and it is not even a required property for these services.

NixOS

Putting NixOS in the name, besides being redundant, would seem to imply that the modules are not portable, which is also not necessarily the case. (though I did shift the goalposts a bit for that second argument)

An intriguing thought is that "subsystem" could replace "service", which seems appropriate when these things encompass more than a process and its execution settings, while also giving space for multiple services to exist within it.
However, "subsystem" hardly gives any intuition for what the feature is for, and the "system" part of "subsystem" conflicts with the system role of these services.
It also reminds me that a current flaw is that process is required, even if you want to just compose a "subsystem" from multiple services instead of a custom process. I guess a nullOr submodule would fix that.

[bew]

About naming, it seems you haven't considered changing the 'services' part 🤔

Systemd base resource names are units not services, and this terminology works for all resource types that systemd supports, like services, timer, target, socket, mount...

So in our case we could have system.units ?

Although this might easily increase the coverage support for types of manageable systemd resources, it might be too much for this proposal 🤔

[ElvishJerricco]

So in our case we could have system.units ?

Similar problem because we do have systemd.units :P

[bjornfor]

So in our case we could have system.units ?

Similar problem because we do have systemd.units :P

systemUnits then?

[roberth]

units

This feature isn't really about systemd units. For instance, the process options are applicable to any init system or process supervisor, and it is possible to write modular services that support all such systems (by sticking to portable options) or supporting specific systems by feature testing (config = ... // optionalAttrs (options?systemd) ...; or other, nicer mechanisms).
Similarly, assertions and warnings are a configuration management facility that goes beyond systemd, but more significantly, so are the capabilities listed by @aanderse.

[7c6f434c]

Does this take any position on configuration files (like a recommended way of integrating with NixOS/rfcs#42 for example)?

[roberth]

The proof of concept uses an option tree that starts with the name of the software, so you'd have options in a service like

• nginx.configFile
• nginx.settings

We could pick a fixed prefix instead, e.g. app => app.settings, app.configFile.
It would still be free-form below app, so not immediately suitable for generic code, but at least it doesn't spoil the namespace with an unbounded set of names that might impede supervisor integrations.

(like a recommended way of integrating with NixOS/rfcs#42 for example)?

42 is great in many contexts, so definitely worth a mention. As for integrating, if we go with app, then app.settings would make for a good default place to put them, certainly if the service has a single config file or a clear "main" config file.

[7c6f434c]

My experience reusing NixOS modules as «modular» services on non-NixOS is that sometimes the config generation is put into a let so usage outside putting the config file in the command line of the start command is blocked. And sometimes there are files put into /etc which can be fished out from /etc generation code but not from a clear place in the module. That's why I think that maybe establishing some kind of best practices on where does the extra options go and where does the config file generation go might help with flexibility / reusability (and thus portability).

[roberth]

@7c6f434c We could make it a convention to put config file stuff in a separate module that doesn't have any process-related options or definitions. This module is then responsible for declaring options like settings and configFile. Definitions for settings would be minimal, and configFile defaults to the generated file containing the settings.
This way you can evalModules or imports it separately and use it basically like a function from settings to configFile.

The main module imports the config file module and adds a generic process definition and/or supervisor-specific definitions, depending which one they're loaded into using (options?some.attr.path) or something fancier.

We don't have something for symlinked config files yet ("etc/"), but that would just mean that instead of configFile it's a standard option with different semantics.

---

app

Having second thoughts.

• weird (it's like the early 00's idea of app?)
• not related to flake apps
• feels like it adds significant noise
• does not seem to enable generic code to use it or anything
• @7c6f434c's use case does not involve generic code that works for more than one service at a time, it seems

[infinisil]

Me and @hsjobeki looked at this together and only had some minor nits. We approve of this direction overall!

We believe this can be merged without an RFC after resolving these discussion points that were brought up:

• Perhaps having a brief guideline to describe best practices for services depending on other services (Modular services #372170 (review))
• A better name for system.services that won't be confused with systemd.services (Modular services #372170 (comment))

We believe these points don't need to be addressed in this PR, but can be follow-up work:

• Automated docs for the service-specific options (brought up in discussion with @hsjobeki)
• Establishing best practices for config generation, "/etc", etc. (Modular services #372170 (comment))

[infinisil]

Suggested change

	imports = [ nixpkgs.modules.services.foo ];
	imports = [ pkgs.foo.services.default ];

[fricklerhandwerk]

That would only make sense if we finally invert the relationship between derivations and attribute sets: Currently derivations have attrs attached, but an application (package) actually has service modules and derivations attached. See ngi-nix/ngipkgs#507 for an explicit implementation of that notion (where we call applications "projects", but that will likely change).

Yes, it's not strictly required to make sense, but it would help a lot.

[roberth]

It also makes sense for a broadly defined package, which has Nix-specific behaviors like outPath, as well as attributes like tests and services that are about using the outputs.

I can not make the introduction of ngipkgs-style applications a prerequisite for this PR, but we could change our conventions away from passthru, into applications when such a change materializes.

[infinisil]

Style nit: Putting this in a variable would improve readability (though also negligibly decrease performance)

[hsjobeki]

I like this change overall pretty much. I have the concern that services are now more difficult to discover since they have (even) more ways to be enabled now. So i hope we understand the significance that is going to be added with this PR and thinking that it really presses on the need of improved discoverability.
But it's also certainly a step in the right direction, where people can find service via search.nixos.org more easily because the package also exports the service (should be shown and searchable). We should improve the discoverability in parallel to allow for such changes. But i wouldn't block on that happening first.

Also adding those services opens a new opportunity of abstracting away from systemd and getting closer to the actual user via the options that each specific service defines. Layered approach ontop of systemd or alternatives. While systemd might be a good interface for computer systems it is not always the best language towards configuring more high level or user oriented services.

[hsjobeki]

What do you think about services.<serviceName>.instances.<instanceName> ?

Pro:
Would allow later introduction of service level attributes within the services.<serviceName> attribute and therefore making it more forward compatible.

Cons:
Choosing an instanceName could be less intuitiv since it is arbitrary

[fricklerhandwerk]

This proposal already allows arbitrary instance names, as noted in #372170 (comment). I suspect doing it the way you propose makes composition via imports impossible. Unless the instances are of type submodule, but then why not do it as already proposed?

[roberth]

@hsjobeki what you call a service in your suggestion is currently a service module (i.e. a Module System module, a file, a deferredModule value, etc).
What you call an instance is a submodule evaluation, which is a Module System configuration (ie evalModules result) turned into an option value.

By using "first class" module values for the services, we avoid having to duplicate this concept as options, and eventually create a bespoke namespacing system to facilitate working with those. Instead, we can use the existing naming and scoping facilities (Nix lexical scope, path literals, Module System module arguments) to handle the logistics around service modules.

This keeps the option tree simple and clean.

[hsjobeki]

This proposal already allows arbitrary instance names, as noted in #372170 (comment). I suspect doing it the way you propose makes composition via imports impossible. Unless the instances are of type submodule, but then why not do it as already proposed?

Yes i saw that. My concern was about mixing two different scopes together into one freeform string.
Those concerns beeing:
service kind and instance name. Currently resulting in something like ghostunnel-plain-old, which could be ghostunnel.instances.plain-old instead. So we acheive a seperation of those concerns. That could maybe also be acheived in a different way. (If even wanted) I had this idea in mind that a serviceName could relate to a packageName such as in the ghostunnel example. This PR already assumes that it is a 1 to 1 relationship, this would only make it more clear.

Let me know what you think.

[fricklerhandwerk]

Yeah I definitely see the clarity in conveying that distinction. But I also see the aspect of keeping the machinery simple. Seems like a bit of a tough nut to crack, and I'd probably need to write and use a bunch of things in one or the other representation to get a feeling for what that's like. It could be that with this design decision we'd be distributing effort between consumers and producers of service modules and configurations, so it's an important concern.

[hsjobeki]

Another concern that also might require it to seperate the Service from the instances is If something needs to be done only once per Service.

For example lets say "ghostunnel" doesnt only set systemd services. But also Firewall Ports or User groups. (Which are lists)

Setting those multiple times on each instance is not good enough. Because we need to 'merge/fold' those together and there is no generic logic for when to deduplictate config that refers 1 to 1 to a Service and not to the instance.

[roberth]

This mechanism is mostly intended for services that can be specified in a modular manner.
Any non-modular aspects are still the responsibility of the user or caller.

For instance, the NixOS module for the OpenSSH service could be factored into two parts: a modular service module that knows things about invoking sshd, and a NixOS module that creates an instance of that module and opens a port for it.

Allocating ports requires a degree of "global" knowledge that makes it tricky to handle it automatically.
As for users, we could provide a semi-portable option that enables the creation of a user, whose name is derived from the instance name/path (ie path when it's a sub-service of another service).
If a different naming scheme for users is required, then that would have to be facilitated by the caller.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/formatting-team-meeting-2025-03-18/61868/1

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/using-hashes-for-stateversion-instead-of-human-readable-strings/61823/38

[aanderse]

i had some motivation to play with this branch on the weekend so i tried to implement a backend for a different service manager than systemd

i guess a little bit of work would need to be done to actually make this code service manager agnostics - my assumptions about this code and making it service manager agnostic are:

• the ghostunnel/service.nix file includes systemd units unconditionally - the entire systemd config should eventually be guarded off by an if statement so there isn't systemd config in my non systemd backend, right?
• the ghostunnel/service.nix uses systemd specific stuff like $CREDENTIALS_DIRECTORY in the generic process.executable option - that should also be guarded off by an if statement, right?

---

overall the process was very simple and took me under 30 minutes to integrate this into the service manager i was using... so i feel fairly positive about this PR now - i will reiterate though, i just wish that someone in the community would implement at least one of my checklist items to show interest... not everything needs to rest on the shoulders of @roberth, right? 😅

[aanderse]

not all service managers provide a direct way to set PATH per service like systemd does

this could also cover environment variables too - which i mention because i have seen a few service managers which support a single environment file which leaves end users without a way to set environment variables if distro upstream uses this file

[ehmry]

I tried modularizing an existing service and found a case where I needed the value of config.security.wrapperDir. The simple solution was to add a securityWrapperDir option to the interface of my module and explicity pass it through, but do we have a better way of doing? Are the specifics of adding fusermount3 to my PATH an artifact of using NixOS and not within the scope of the portable module?

07383b4

[dwt]

I still haven't really understood the implementation of this, but I could imagine that this should allow me to implement a provider, which deploys a systemd service on a non nixos linux system.

Would that work? I'd love to give it a try.

[aanderse]

Would that work? I'd love to give it a try.

yes, you could do that. please keep us informed 👍

[hsjobeki]

I tried modularizing an existing service and found a case where I needed the value of config.security.wrapperDir. The simple solution was to add a securityWrapperDir option to the interface of my module and explicity pass it through, but do we have a better way of doing? Are the specifics of adding fusermount3 to my PATH an artifact of using NixOS and not within the scope of the portable module?

I see this PR as a first step. It includes the minimal feature-set which happen to be systemd units.
If additional features are needed. Such as:

@ehmry

securityWrapperDir

or

@AndersE

create system users
place files under /etc
set runtime parameters of the linux kernel, as set by sysctl
add packages to the global system environment
open ports in the system firewall

We could add these options in future PRs.
Although there should be some limit about the forwarded options, because it could get confusing rather quickly, how these things are mapped under the hood. Good documentation would help a lot.

[roberth]

The simple solution was to add a securityWrapperDir option to the interface of my module and explicity pass it through, but do we have a better way of doing?

I can imagine making the whole NixOS config available for reading in the service module, with the caveat that it's only available on NixOS.

Are the specifics of adding fusermount3 to my PATH an artifact of using NixOS and not within the scope of the portable module?

As for adding semi-arbitrary things to the NixOS config, I'd say out of scope.
If we find a satisfactory framework for handling such things, we could add it to the scope of modular services, but let's keep it simple for now.

[ehmry]

It's awkward that system.services is defined in a systemd specific module and the portable services are defined by importing systemd/service.nix -> portable/service.nix. Modules for other service-managers need to be imported here, as far as I can tell. But I think it's fine for now.

[roberth]

A different service manager for system.services would imply either

• NixOS offers the choice of a different init system
• or you're using another service manager in addition to systemd

If it's about a different init system, the type of system.services could be configured with an option ("dependent type" - trivial in the module system: type = config.foo;). Also if it's not about NixOS, e.g. nix-darwin can have its own system.services implementation, and I see little opportunity to share code from this file with a nix-darwin provider implementation. (As opposed to the portable/ files of course)

If it's for an additional service manager, it should have its own option instead, which isn't system.services.