[BUG] Jibri server not responding without having active session

[krishnakingdom]

What happened?

I have multiple Jibri servers, those are working fine, but some time Jibri is not recording the session without having the any active session in all jibri servers. When I restart the services in all jibri servers then the recording will start again, like this happened many times, is there any particular reason for this all how could I debug this issue.

Note : I have enabled the Auto Recording option.

This is the jibri version I am using

Platform

• Chrome (or Chromium based)
• Firefox
• Safari
• Other desktop browser
• Android browser
• iOS browser
• Electron app
• Android mobile app
• iOS mobile app
• Custom app using a mobile SDK

Browser / app / sdk version

131.0.6778.205

Relevant log output

No response

Reproducibility

• The problem is reproducible on meet.jit.si

More details?

JIbri Version : 8.0-160-g5af7dd7-1
Running on ubuntu servers with multiple Jibri servers on auto scaling mode.

[krishnakingdom]

Please find the below logs for reference.

jibri-log2.0.txt
jibri-log.0.txt

please check this systemd logs also

-- Logs begin at Tue 2024-02-06 12:34:42 UTC, end at Thu 2025-01-09 06:12:48 UTC. --
Jan 08 11:24:48 jibri-vmsXVCD1E systemd[1]: Stopping Jibri Process...
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: Exception in thread "KtorShutdownHook" java.util.concurrent.TimeoutException
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1892)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2027)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at org.eclipse.jetty.server.Server.doStop(Server.java:490)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.jetty.jakarta.JettyApplicationEngineBase.stop(JettyApplicationEngineBase.kt:75)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.jetty.jakarta.JettyApplicationEngineBase$start$1.invoke(JettyApplicationEngineBase.kt:49)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.jetty.jakarta.JettyApplicationEngineBase$start$1.invoke(JettyApplicationEngineBase.kt:48)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.engine.ShutdownHook.run(ShutdownHookJvm.kt:41)
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: jibri.service: Main process exited, code=exited, status=255/n/a
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: jibri.service: Failed with result 'exit-code'.
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: Stopped Jibri Process.
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: Starting Jibri Process...
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: root : TTY=unknown ; PWD=/ ; USER=jibri ; COMMAND=/usr/bin/google-chrome --timeout=1000 --headless about:blank
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: pam_unix(sudo:session): session opened for user jibri by (uid=0)
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: [0108/112454.511996:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: [0108/112454.517422:WARNING:sandbox_linux.cc(393)] InitializeSandbox() called with multiple threads in process gpu-process.
Jan 08 11:24:55 jibri-vmsXVCD1E sudo[9962]: pam_unix(sudo:session): session closed for user jibri
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9980 (cat) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9981 (cat) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9983 (chrome_crashpad) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9990 (chrome) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9991 (chrome) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: Started Jibri Process.
Jan 08 11:24:56 jibri-vmsXVCD1E launch.sh[10058]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Jan 08 11:24:56 jibri-vmsXVCD1E launch.sh[10058]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Jan 08 11:24:56 jibri-vmsXVCD1E launch.sh[10058]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 24608
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 2571
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 9918
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 23870
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 16436
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 21942
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 12768
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.

ubuntu@jibri-vmsXVCD1E:~$ sudo journalctl -u jibri.service -xe --since "2025-01-08"
-- Logs begin at Tue 2024-02-06 12:34:42 UTC, end at Thu 2025-01-09 06:16:56 UTC. --
Jan 08 11:24:48 jibri-vmsXVCD1E systemd[1]: Stopping Jibri Process...
-- Subject: Unit jibri.service has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

-- Unit jibri.service has begun shutting down.
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: Exception in thread "KtorShutdownHook" java.util.concurrent.TimeoutException
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1892)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2027)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at org.eclipse.jetty.server.Server.doStop(Server.java:490)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.jetty.jakarta.JettyApplicationEngineBase.stop(JettyApplicationEngineBase.kt:75)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.jetty.jakarta.JettyApplicationEngineBase$start$1.invoke(JettyApplicationEngineBase.kt:49)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.jetty.jakarta.JettyApplicationEngineBase$start$1.invoke(JettyApplicationEngineBase.kt:48)
Jan 08 11:24:54 jibri-vmsXVCD1E launch.sh[2274]: at io.ktor.server.engine.ShutdownHook.run(ShutdownHookJvm.kt:41)
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: jibri.service: Main process exited, code=exited, status=255/n/a
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: jibri.service: Failed with result 'exit-code'.
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: Stopped Jibri Process.
-- Subject: Unit jibri.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

-- Unit jibri.service has finished shutting down.
Jan 08 11:24:54 jibri-vmsXVCD1E systemd[1]: Starting Jibri Process...
-- Subject: Unit jibri.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

-- Unit jibri.service has begun starting up.
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: root : TTY=unknown ; PWD=/ ; USER=jibri ; COMMAND=/usr/bin/google-chrome --timeout=1000 --headless about:blank
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: pam_unix(sudo:session): session opened for user jibri by (uid=0)
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: [0108/112454.511996:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.
Jan 08 11:24:54 jibri-vmsXVCD1E sudo[9962]: [0108/112454.517422:WARNING:sandbox_linux.cc(393)] InitializeSandbox() called with multiple threads in process gpu-process.
Jan 08 11:24:55 jibri-vmsXVCD1E sudo[9962]: pam_unix(sudo:session): session closed for user jibri
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9980 (cat) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9981 (cat) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9983 (chrome_crashpad) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9990 (chrome) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: jibri.service: Found left-over process 9991 (chrome) in control group while starting unit. Ignoring.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jan 08 11:24:55 jibri-vmsXVCD1E systemd[1]: Started Jibri Process.
-- Subject: Unit jibri.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support

-- Unit jibri.service has finished starting up.

-- The start-up result is RESULT.
Jan 08 11:24:56 jibri-vmsXVCD1E launch.sh[10058]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Jan 08 11:24:56 jibri-vmsXVCD1E launch.sh[10058]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Jan 08 11:24:56 jibri-vmsXVCD1E launch.sh[10058]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 24608
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:25:10 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 2571
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:26:01 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 9918
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:27:04 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 23870
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 11:27:29 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 16436
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 12:05:53 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 21942
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 08 13:07:58 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: Starting ChromeDriver 114.0.5735.16 (7e1ff058633f5b79b1cd7479aca585ba385519d8-refs/branch-heads/5735@{#182}) on port 12768
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: Only local connections are allowed.
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Jan 09 06:11:47 jibri-vmsXVCD1E launch.sh[10058]: ChromeDriver was started successfully.
lines 29-84/84 (END)

[damencho]

but some time Jibri is not recording the session without having the any active session in all jibri servers.

What do you mean by that?

[krishnakingdom]

I mean when there are no active recordings on jibri server it should allow for the new session recording, but in my scenario since there is no active session it is still not allow to record a session.
I think there is some service is failing so until unless we initiate a Jibri systemd service restart it is not allowing any new session to record.

[damencho]

As you are using the 'Auto Recording' (which is not maintained by us), you need to debug that and see why it is failing.