OBS rapid disconnects/reconnects

[hernanrz]

We've been having this issue for a while now where some users experience disconnects/reconnects in OBS somewhat consistently. We've ruled out network issues on the user's end, also tried different bitrate settings and it doesn't seem to affect things. We're wondering if it's possible a bug in OME is the cause.

About our server setup

• We have multiple concurrent streams
• We use the API to record every stream that starts and call the API to stop recording when the stream disconnects (and stays disconnected for up to 5 seconds)
• We're using the latest OME version

Config file
Server.xml.txt

OBS Logs from the disconnects
obslog.txt

[getroot]

I haven't experienced this yet. Are you using the latest release of OvenMediaEngine?
I'm not familiar with OBS logs, but I'm seeing a lot of error logs that are different from RTMP streams.

[naanlizard]

Some extra data

Dockerfile

ARG CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX

# https://hub.docker.com/layers/airensoft/ovenmediaengine/dev/images/sha256-08518b7cd9eccaada9faed318765b1fbbda78dad0805552e3eb039e21e563300
FROM $CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX/airensoft/ovenmediaengine@sha256:f5d138ee90c3e3206708f95face1ca7ca9dae2818ef532fc743d280b4c346f1a

# UID/username matching
RUN groupadd -g 1000 user && \
    useradd -m -u 1000 -g user user

USER user

WORKDIR /opt/ovenmediaengine/bin

COPY --chown=1000:1000 ./Server.xml /opt/ovenmediaengine/bin/origin_conf/Server.xml
COPY --chown=1000:1000 ./Logger.xml /opt/ovenmediaengine/bin/origin_conf/Logger.xml

# We don't use the origin/edge split, just the one instance which is the origin
#COPY --chown=1000:1000 ./EdgeConfig.xml /opt/ovenmediaengine/bin//edge_conf/Server.xml

HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 CMD curl --fail -H "Authorization: Basic $(printf ${API_ACCESS_TOKEN} | base64)"  'http://localhost:48081/v1/vhosts' 

CMD ["/opt/ovenmediaengine/bin/OvenMediaEngine", "-c", "origin_conf"]

Logger.xml - increased log level for http (we were having some issues with nginx proxying, keepalives, and screenshots, which I can't recall if I've posted about already, if not I will soon)

<?xml version="1.0" encoding="UTF-8"?>

<Logger version="2">
<!-- Log file location -->
<Path>/var/log/ovenmediaengine</Path>

<!-- Disable some SRT internal logs -->
<Tag name="SRT" level="critical" />
<Tag name="HttpServer" level="debug" />
<!--<Tag name="RTMPProvider.Stat" level="info" /> -->
<Tag name=".*\.Stat" level="warn" />

<!-- Log level: [debug, info, warn, error, critical] -->
<Tag name=".*" level="debug" />
</Logger>

Screenshots from grafana, showing the clients and incoming bitrate reported by OME (via oven-exporter) for the stream that was logged, in the OP comment.

OME logs (partially redacted, loki query follows - {container_name=~"hostname-ome-production"} != [Request] != orchestrator.cpp != Try to pull stream != Could not find Origin != Cannot find stream != HTTP.Server != longer-than-expected != [Response] != https://playback.hostname.tld != HttpConnection != stream_metrics != AVCodecContext != http_connection != http_server != AVFilter |= ``)

Explore-logs-2025-02-20 13_52_11.txt

And a separate OBS log if it helps, of a more severe instance a while back, which we don't have OME logs for unfortunately

bzUMmjOc8RSBHtLP.txt

It's possible the CORS errors are related, we're investigating those separately. But it seemed prudent to bring it up here in hopes of debug help regarding OME

[getroot]

I have some questions to analyze this problem.

• Is a new RTMP input not being created? Are existing RTMP inputs still maintained? Or are all RTMP connections disconnected?
• How often have you experienced this issue?
• Does it recover automatically over time, or do you restart the server?

[naanlizard]

This is an infrequent (a few times a week perhaps?) problem that has been plaguing us for at least a year now, as you can see from the second OBS log I posted

After a few minutes (up to 30 minutes at worst, but usually not that bad) the situation stabilizes

It appears to start with one RTMP connection being closed, and OBS trying to reconnect, which fails - according to these logs, apparently because it would be a duplicate stream. Here are some OME and our backend logs that I just found

ome logs.txt
backend logs.txt

We do have debug logging enabled as seen in logger.xml, so let me know if I should un-filter some of that

[getroot]

Thanks for the detailed information. This is a really hard issue to find the root cause of. I'll have to get our developers together and discuss this in depth.

[naanlizard]

Some better logs, very clearly showing the issue. If you tell me in the next few days, I can share more complete logs if necessary - this has a few things like fmp4 and keyframe interval and b frame notifications removed. Also the user's obs log file

The second file is both our backend and OME logs in one file, which may be easier to read?

OBS log file.txt

Explore-logs-2025-02-21 22_48_11.txt

[getroot]

One more question for troubleshooting.

To briefly explain the issue,

1. Sometimes someone fails to connect to OME using OBS.

2. This resolves itself within a few minutes.

3. This appears to be a duplicate stream name when looking at the logs.

What about other sessions when the issue occurs? Do you experience issues with existing OBS connections being disconnected or LLHLS playback failing?

[naanlizard]

Usually I think the stream is already connected, then disconnects for some reason, and reconnects (or fails to) Other streams are not affected

…

On Mon, Feb 24, 2025 at 12:04 Jeheon Han ***@***.***> wrote: One more question for troubleshooting. To briefly explain the issue, 1. Sometimes someone fails to connect to OME using OBS. 2. This resolves itself within a few minutes. 3. This appears to be a duplicate stream name when looking at the logs. What about other sessions when the issue occurs? Do you experience issues with existing OBS connections being disconnected or LLHLS playback failing? — Reply to this email directly, view it on GitHub <#1782 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWL2HHGGOV4OJKDMUBD2RMDGPAVCNFSM6AAAAABXPW6EZ2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMRZHE4TSNI> . You are receiving this because you commented.Message ID: ***@***.*** com>

[getroot]

Thank you. If it doesn't affect other streams, the cause of the issue becomes more narrowed down. Thank you.

[naanlizard]

We are speculating that this issue is exacerbated by a memory leak in a browser source component of obs We've pushed a patch to our chat which should improve performance and stop the memory leak, we'll see if we continue getting flickering. I think the relevant part for ome, is just that when obs disconnects, it should immediately be able to reconnect with ome. I hope that's clear

…

On Mon, Feb 24, 2025 at 12:13 Jeheon Han ***@***.***> wrote: Thank you. If it doesn't affect other streams, the cause of the issue becomes more narrowed down. Thank you. — Reply to this email directly, view it on GitHub <#1782 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV4BWO6SRXDUVWBBY7U7CL2RMEH7AVCNFSM6AAAAABXPW6EZ2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMZQGAYDMNQ> . You are receiving this because you commented.Message ID: ***@***.*** com>

[getroot]

Moved to the Issue board.
#1789