services: add keycloak

[gabyx]

• Add keycloak service.
• Add optional realm import and export.
• Add example.
• Test that it works.
• Make health prope script work.

@andrzejressel, @tboerger : I try to continue your effort to make it work.

Notes:
I have a docker-compose file which works so far:
https://gitlab.com/data-custodian/custodian/-/blob/beea86076aef53ea00d7bf2aca7832d237022ac4/deploy/docker/docker-compose.yaml#L352

[tboerger]

Looks already quite good, I will try to figure out a working configuration. Some more options like ports would be great and maybe even some env config to add more env variables.

[gabyx]

@tboerger: Port is already there. Jeah Env config seems good!

[tboerger]

Yeah but there are ports for http, https and internal.

[gabyx]

I think I cannot make it work. Keycloak tries to write (I assume) (it builds itself?) to the Nix store when launched, or to another write only directory.

I am not so familiar what the "$exeDir/bin/kc.sh" --verbose start-dev actually does.

I always get: some dumb JAVA build failures.

[keycloak       ] ERROR: Failed to run 'build' command.
[keycloak       ] ERROR: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[keycloak       ]       [error]: Build step io.quarkus.deployment.pkg.steps.JarResultBuildStep#buildRunnerJar threw an exception: java.nio.file.ReadOnlyFileSystemException
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.checkWritable(ZipFileSystem.java:370)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.createDirectory(ZipFileSystem.java:708)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipPath.createDirectory(ZipPath.java:742)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.createDirectory(ZipFileSystemProvider.java:186)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectory(Files.java:699)
[keycloak       ]       at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectories(Files.java:752)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildThinJar(JarResultBuildStep.java:664)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildRunnerJar(JarResultBuildStep.java:228)
[keycloak       ]       at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
[keycloak       ]       at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
[keycloak       ]       at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
[keycloak       ]       at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
[keycloak       ]       at java.base/java.lang.Thread.run(Thread.java:1583)
[keycloak       ]       at org.jboss.threads.JBossThread.run(JBossThread.java:483)
[keycloak       ]
[keycloak       ] ERROR: Build failure: Build failed due to errors
[keycloak       ]       [error]: Build step io.quarkus.deployment.pkg.steps.JarResultBuildStep#buildRunnerJar threw an exception: java.nio.file.ReadOnlyFileSystemException
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.checkWritable(ZipFileSystem.java:370)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.createDirectory(ZipFileSystem.java:708)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipPath.createDirectory(ZipPath.java:742)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.createDirectory(ZipFileSystemProvider.java:186)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectory(Files.java:699)
[keycloak       ]       at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectories(Files.java:752)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildThinJar(JarResultBuildStep.java:664)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildRunnerJar(JarResultBuildStep.java:228)
[keycloak       ]       at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
[keycloak       ]       at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
[keycloak       ]       at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
[keycloak       ]       at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
[keycloak       ]       at java.base/java.lang.Thread.run(Thread.java:1583)
[keycloak       ]       at org.jboss.threads.JBossThread.run(JBossThread.java:483)
[keycloak       ]
[keycloak       ] For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.

@talyz: Do you might know what the problem is here since you have hopefully some insight. Does the NixOS modules do something different? https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/keycloak.nix#L766

I just try to launch it in http only mode: https://github.com/cachix/devenv/pull/1806/files#diff-d8696b88eb47713cf77037d6a38f2ddeb99a11d2256064160d2aaa67ca6b36e1R133

[tboerger]

Have you tried start instead if start-dev?

[gabyx]

@tboerger : yes, I think I need to run another build by overriding the configFile like here

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/keycloak.nix#L503

so we need to make a keycloak.conf settings file and then build keycloak and the use that.
I am unsure if we should use the same settings
or just write a simple keycloak.conf settings file.

@tboerger : If you could figure out how that would look like, I could test it tomorrow.

[talyz]

Yes, you do indeed need to provide the config as is done in the NixOS module, since quarkus will try to rebuild its image if the config doesn't match its cached one.

[gabyx]

@sandydoo: Is is actually possible to reuse (not sure how?) NixOS Modules and is the module system of devenv the same as NixOS's one?
Or should we just copy&pasta&adapt the stuff for a devenv service, probably because we do not want certain things like in the real service etc? Whats the advice here?

@tboerger: I will first try to make dev-mem db settings work, and then when we have a working thing, we could add the postgress stuff maybe by enabling automatically the prostgres service and using this instance etc... but slowely.

[gabyx]

@talyz: Kudos for your module, its great!

[tboerger]

Since this is for dev environments I would prefer to keep it simple without postgres and just use the builtin file-based database. Was it sqlite or h2 by default?

[gabyx]

not sure I think its dev-mem or dev-file you can choose from

[tboerger]

If you simply skip the env variable for the database driver it should be fine.

[gabyx]

@tboerger: I dont have time next week to test if it starts up. So far the thing should build, maybe you can give it a test,

direnv allow # or nix develop 
./result/bin/devenv-run-tests --only keycloak

if it starts up or not. Maybe propose changes I can integrate after 14. April.
I added also (not tested yet) a process which exports realms, really needed to make changes and keep it presistent.

[gabyx]

@talyz: Do you mind maybe helping out with some questions: I was hoping to get keycloak up and running with db = dev-file or dev-mem but I guess these option are always overridden by the derivation build. Do you know how I can build it with dev-mem or dev-file?
I cannot start it unfortunately with kc.sh start-dev cause that always builds the stuff (?).
I see that keycloak passes a --profile=dev flag in https://github.com/keycloak/keycloak/blob/main/quarkus/dist/src/main/content/bin/kc.sh#L70
Probably I need to pass this to the build ?

[gabyx]

@talyz: Do you mind maybe helping out with some questions: I was hoping to get keycloak up and running with db = dev-file or dev-mem but I guess these option are always overridden by the derivation build. Do you know how I can build it with dev-mem or dev-file?

I cannot start it unfortunately with kc.sh start-dev cause that always builds the stuff (?).

I see that keycloak passes a --profile=dev flag in https://github.com/keycloak/keycloak/blob/main/quarkus/dist/src/main/content/bin/kc.sh#L70

Probably I need to pass this to the build ?

It works now: I forgot to symlink some folders which are apparently used for the build, that was quite a hassle. I try to make the export/import of the realms also work...

[gabyx]

@tboerger: The PR is finished and supports now everything I want.

• Starting Keycloak properly with DB backends dev-file, and dev-mem (export here does not work as it runs in memory, import however works).
• Import Realms.
• Export Realms with process keycloak-realm-export-all which exports all configured realms.
  You can start it with process-manager process start keycloak-realm-export-all -u "$PC_SOCKET_PATH" or with script:
  keycloak-realm-export-all.
• Tested connection, health-check, and import/export.

Example:

services.keycloak = {
  enable = true;
  settings.http-port = 8089;

  database.type = "dev-file";

  realms = {
    master = {
      path = "./realms/master.json"; # Relative to `DEVENV_ROOT`
      export = true;
      import = false;
    };
    test = {
      path = "./realms/test.json";
      export = true;
    };
  };
};

@domenkozar : Thanks for a review.

[tboerger]

I haven't checked the args of the export command, but does this also include secrets? That would help to have some reproducible setup for the development of apps against keycloak.

[gabyx]

I haven't checked the args of the export command, but does this also include secrets? That would help to have some reproducible setup for the development of apps against keycloak.

Yes that includes secrets and everything. Thats exactly what we want as well, so everything is deterministic if you reimport it.

[gabyx]

@domenkozar: How can I request a review from maintainers? Would be really good, because the feature would help already to be in devenv. =)

[gabyx]

@tboerger, @domenkozar: The PR has been rebased, and another test was added for sslCertificates.
Thanks for merging. (Check the last two commits).

[domenkozar]

Thank you for all this work @gabyx :) Due to GitHub Actions limitations we've hit 250 jobs limit, could I ask you to move keycloack from examples to tests? We're going to fix this in the future but for now this is the only option we have.

[gabyx]

Jeah, I try. If I understand them =).

[tboerger]

I don't want to annoy the developers which don't need any realm export, is there a way I can get rid of the keycloak-realm-export-all process within the process list?

[sandydoo]

I don't want to annoy the developers which don't need any realm export, is there a way I can get rid of the keycloak-realm-export-all process within the process list?

It looks like it should be disabled if no realms have export = true set. Are you seeing something different?

[tboerger]

I got a single realm with import true and export false

[gabyx]

I can add a fix for that, check tomorrow probably a simple switch.

[tboerger]

The twisted username/password fix is already merged. But an option to hide the export process within the process manager would be great :)

[gabyx]

@tboerger : I add a fix for it.

[gabyx]

@tboerger : #1947