feat: add keycloak service #1806

gabyx · 2025-04-02T14:45:27Z

Add keycloak service.
Add optional realm import and export.
Add example.
Test that it works.
Make health prope script work.

@andrzejressel, @tboerger : I try to continue your effort to make it work.

Notes:
I have a docker-compose file which works so far:
https://gitlab.com/data-custodian/custodian/-/blob/beea86076aef53ea00d7bf2aca7832d237022ac4/deploy/docker/docker-compose.yaml#L352

tboerger · 2025-04-02T14:57:02Z

Looks already quite good, I will try to figure out a working configuration. Some more options like ports would be great and maybe even some env config to add more env variables.

gabyx · 2025-04-02T15:01:03Z

@tboerger: Port is already there. Jeah Env config seems good!

tboerger · 2025-04-02T15:02:41Z

Yeah but there are ports for http, https and internal.

gabyx · 2025-04-02T16:56:31Z

I think I cannot make it work. Keycloak tries to write (I assume) (it builds itself?) to the Nix store when launched, or to another write only directory.

I am not so familiar what the "$exeDir/bin/kc.sh" --verbose start-dev actually does.

I always get: some dumb JAVA build failures.

[keycloak       ] ERROR: Failed to run 'build' command.
[keycloak       ] ERROR: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[keycloak       ]       [error]: Build step io.quarkus.deployment.pkg.steps.JarResultBuildStep#buildRunnerJar threw an exception: java.nio.file.ReadOnlyFileSystemException
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.checkWritable(ZipFileSystem.java:370)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.createDirectory(ZipFileSystem.java:708)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipPath.createDirectory(ZipPath.java:742)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.createDirectory(ZipFileSystemProvider.java:186)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectory(Files.java:699)
[keycloak       ]       at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectories(Files.java:752)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildThinJar(JarResultBuildStep.java:664)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildRunnerJar(JarResultBuildStep.java:228)
[keycloak       ]       at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
[keycloak       ]       at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
[keycloak       ]       at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
[keycloak       ]       at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
[keycloak       ]       at java.base/java.lang.Thread.run(Thread.java:1583)
[keycloak       ]       at org.jboss.threads.JBossThread.run(JBossThread.java:483)
[keycloak       ]
[keycloak       ] ERROR: Build failure: Build failed due to errors
[keycloak       ]       [error]: Build step io.quarkus.deployment.pkg.steps.JarResultBuildStep#buildRunnerJar threw an exception: java.nio.file.ReadOnlyFileSystemException
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.checkWritable(ZipFileSystem.java:370)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.createDirectory(ZipFileSystem.java:708)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipPath.createDirectory(ZipPath.java:742)
[keycloak       ]       at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.createDirectory(ZipFileSystemProvider.java:186)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectory(Files.java:699)
[keycloak       ]       at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807)
[keycloak       ]       at java.base/java.nio.file.Files.createDirectories(Files.java:752)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildThinJar(JarResultBuildStep.java:664)
[keycloak       ]       at io.quarkus.deployment.pkg.steps.JarResultBuildStep.buildRunnerJar(JarResultBuildStep.java:228)
[keycloak       ]       at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
[keycloak       ]       at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
[keycloak       ]       at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
[keycloak       ]       at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
[keycloak       ]       at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
[keycloak       ]       at java.base/java.lang.Thread.run(Thread.java:1583)
[keycloak       ]       at org.jboss.threads.JBossThread.run(JBossThread.java:483)
[keycloak       ]
[keycloak       ] For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.

@talyz: Do you might know what the problem is here since you have hopefully some insight. Does the NixOS modules do something different? https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/keycloak.nix#L766

I just try to launch it in http only mode: https://github.com/cachix/devenv/pull/1806/files#diff-d8696b88eb47713cf77037d6a38f2ddeb99a11d2256064160d2aaa67ca6b36e1R133

tboerger · 2025-04-02T17:03:48Z

Have you tried start instead if start-dev?

gabyx · 2025-04-02T17:11:11Z

@tboerger : yes, I think I need to run another build by overriding the configFile like here

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/web-apps/keycloak.nix#L503

so we need to make a keycloak.conf settings file and then build keycloak and the use that.
I am unsure if we should use the same settings
or just write a simple keycloak.conf settings file.

@tboerger : If you could figure out how that would look like, I could test it tomorrow.

talyz · 2025-04-02T20:08:40Z

Yes, you do indeed need to provide the config as is done in the NixOS module, since quarkus will try to rebuild its image if the config doesn't match its cached one.

gabyx · 2025-04-03T05:08:07Z

@sandydoo: Is is actually possible to reuse (not sure how?) NixOS Modules and is the module system of devenv the same as NixOS's one?
Or should we just copy&pasta&adapt the stuff for a devenv service, probably because we do not want certain things like in the real service etc? Whats the advice here?

@tboerger: I will first try to make dev-mem db settings work, and then when we have a working thing, we could add the postgress stuff maybe by enabling automatically the prostgres service and using this instance etc... but slowely.

gabyx · 2025-04-03T05:35:35Z

@talyz: Kudos for your module, its great!

tboerger · 2025-04-03T06:57:51Z

Since this is for dev environments I would prefer to keep it simple without postgres and just use the builtin file-based database. Was it sqlite or h2 by default?

gabyx · 2025-04-03T17:27:30Z

not sure I think its dev-mem or dev-file you can choose from

tboerger · 2025-04-03T18:54:48Z

If you simply skip the env variable for the database driver it should be fine.

gabyx · 2025-04-03T19:55:59Z

@tboerger: I dont have time next week to test if it starts up. So far the thing should build, maybe you can give it a test,

direnv allow # or nix develop 
./result/bin/devenv-run-tests --only keycloak

if it starts up or not. Maybe propose changes I can integrate after 14. April.
I added also (not tested yet) a process which exports realms, really needed to make changes and keep it presistent.

gabyx · 2025-04-15T11:27:37Z

@talyz: Do you mind maybe helping out with some questions: I was hoping to get keycloak up and running with db = dev-file or dev-mem but I guess these option are always overridden by the derivation build. Do you know how I can build it with dev-mem or dev-file?
I cannot start it unfortunately with kc.sh start-dev cause that always builds the stuff (?).
I see that keycloak passes a --profile=dev flag in https://github.com/keycloak/keycloak/blob/main/quarkus/dist/src/main/content/bin/kc.sh#L70
Probably I need to pass this to the build ?

gabyx · 2025-04-15T17:11:22Z

@talyz: Do you mind maybe helping out with some questions: I was hoping to get keycloak up and running with db = dev-file or dev-mem but I guess these option are always overridden by the derivation build. Do you know how I can build it with dev-mem or dev-file?

I cannot start it unfortunately with kc.sh start-dev cause that always builds the stuff (?).

I see that keycloak passes a --profile=dev flag in https://github.com/keycloak/keycloak/blob/main/quarkus/dist/src/main/content/bin/kc.sh#L70

Probably I need to pass this to the build ?

It works now: I forgot to symlink some folders which are apparently used for the build, that was quite a hassle. I try to make the export/import of the realms also work...

- `dev-mem` does not work, as it runs in memory and `kc.sh export` is a single shot command needing `dev-file`.

gabyx · 2025-04-16T14:43:59Z

@tboerger: The PR is finished and supports now everything I want.

Starting Keycloak properly with DB backends dev-file, and dev-mem (export here does not work as it runs in memory, import however works).
Import Realms.
Export Realms with process keycloak-realm-export-all which exports all configured realms.
You can start it with process-manager process start keycloak-realm-export-all -u "$PC_SOCKET_PATH" or with script:
keycloak-realm-export-all.
Tested connection, health-check, and import/export.

Example:

services.keycloak = {
  enable = true;
  settings.http-port = 8089;

  database.type = "dev-file";

  realms = {
    master = {
      path = "./realms/master.json"; # Relative to `DEVENV_ROOT`
      export = true;
      import = false;
    };
    test = {
      path = "./realms/test.json";
      export = true;
    };
  };
};

@domenkozar : Thanks for a review.

tboerger

I haven't checked the args of the export command, but does this also include secrets? That would help to have some reproducible setup for the development of apps against keycloak.

tboerger · 2025-04-16T16:32:39Z

src/modules/services/keycloak.nix

+
+    database = {
+      type = mkOption {
+        type = types.enum [


Since it would also work with postgres or mariadb does this enum really make sense? By default it's anyway using a file-based database which should be stored in the devenv state path.

so far we only support dev-mem and dev-file thats why I only added those, the otherones need more complex setup etc etc. Does that make sense?

gabyx · 2025-04-17T07:45:38Z

I haven't checked the args of the export command, but does this also include secrets? That would help to have some reproducible setup for the development of apps against keycloak.

Yes that includes secrets and everything. Thats exactly what we want as well, so everything is deterministic if you reimport it.

feat: add first initial files for keycloak

cd1ade6

gabyx marked this pull request as draft April 2, 2025 14:45

gabyx changed the title ~~feat: add first initial files for keycloak~~ feat: add keycloak service Apr 2, 2025

gabyx added 6 commits April 2, 2025 17:08

fix: types

a277722

fix: remove obsolete file

33c1dfc

fix: correct some config mistakes

58009aa

fix: stringigy error (?)

cf779b6

fix: dummy test

18bac66

fix: improve some tests

8399ecd

gabyx force-pushed the feat/add-keycloak-service branch from 467ab89 to 8399ecd Compare April 2, 2025 16:46

fix: try to copy keycloak to local fodler to make startup work

2f9defd

fix: adapt to NixOS keycloak module

3bbd02b

fix: correct mistakes on submodule

929806e

gabyx added 3 commits April 3, 2025 21:20

fix: add realm export process to start manually

817f48c

fix: typos

09ac20c

fix: mkIf for realm export

f62c0e1

gabyx force-pushed the feat/add-keycloak-service branch from 968aaa8 to f62c0e1 Compare April 3, 2025 19:49

fix: more testing...

b4bb1fb

gabyx mentioned this pull request Apr 15, 2025

keycloak: add extra build args for smoother overriding NixOS/nixpkgs#398850

Draft

13 tasks

fix: make keycloak startup

d84bfe3

gabyx force-pushed the feat/add-keycloak-service branch from 6185969 to d84bfe3 Compare April 15, 2025 15:02

fix: realm export name

2659a57

gabyx force-pushed the feat/add-keycloak-service branch from 2106f5e to 2659a57 Compare April 15, 2025 15:40

fix: realm export (WIP)

71f161b

fix: fix tests with import and export

fa65f76

- `dev-mem` does not work, as it runs in memory and `kc.sh export` is a single shot command needing `dev-file`.

gabyx marked this pull request as ready for review April 16, 2025 14:40

tboerger reviewed Apr 16, 2025

View reviewed changes

fix: small corrections

5616b92

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add keycloak service #1806

feat: add keycloak service #1806

gabyx commented Apr 2, 2025 •

edited

Loading

tboerger commented Apr 2, 2025

gabyx commented Apr 2, 2025

tboerger commented Apr 2, 2025

gabyx commented Apr 2, 2025

tboerger commented Apr 2, 2025

gabyx commented Apr 2, 2025

talyz commented Apr 2, 2025

gabyx commented Apr 3, 2025

gabyx commented Apr 3, 2025

tboerger commented Apr 3, 2025

gabyx commented Apr 3, 2025

tboerger commented Apr 3, 2025

gabyx commented Apr 3, 2025

gabyx commented Apr 15, 2025

gabyx commented Apr 15, 2025

gabyx commented Apr 16, 2025 •

edited

Loading

tboerger left a comment

tboerger Apr 16, 2025

gabyx Apr 17, 2025

gabyx commented Apr 17, 2025

feat: add keycloak service #1806

Are you sure you want to change the base?

feat: add keycloak service #1806

Conversation

gabyx commented Apr 2, 2025 • edited Loading

tboerger commented Apr 2, 2025

gabyx commented Apr 2, 2025

tboerger commented Apr 2, 2025

gabyx commented Apr 2, 2025

tboerger commented Apr 2, 2025

gabyx commented Apr 2, 2025

talyz commented Apr 2, 2025

gabyx commented Apr 3, 2025

gabyx commented Apr 3, 2025

tboerger commented Apr 3, 2025

gabyx commented Apr 3, 2025

tboerger commented Apr 3, 2025

gabyx commented Apr 3, 2025

gabyx commented Apr 15, 2025

gabyx commented Apr 15, 2025

gabyx commented Apr 16, 2025 • edited Loading

tboerger left a comment

Choose a reason for hiding this comment

tboerger Apr 16, 2025

Choose a reason for hiding this comment

gabyx Apr 17, 2025

Choose a reason for hiding this comment

gabyx commented Apr 17, 2025

gabyx commented Apr 2, 2025 •

edited

Loading

gabyx commented Apr 16, 2025 •

edited

Loading