Conflict between prometheus-cpp and NVIDIA DeepStream REST server #743
Description
Description:
I’ve encountered a conflict when using prometheus-cpp alongside the NVIDIA DeepStream REST/RTSP server. When Prometheus is enabled in my project, the DeepStream server fails with the following error:
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Aborted (core dumped)
NVIDIA DeepStream nvmultiurisrcbin plugin uses DeepStream REST API Sever.
Disabling Prometheus in the project resolves the issue, and the DeepStream REST server runs normally. This strongly suggests a runtime conflict between prometheus-cpp and DeepStream.
Issue in nvidia forum: DeepStream nvmultiurisrcbin Aborted (core dumped) when adding stream via REST API
Environment:
- DeepStream version: 7.1
- Docker container:
nvcr.io/nvidia/deepstream_7.1-triton-multiarch - OS: Ubuntu (inside NVIDIA DeepStream docker image)
Prometheus headers in use:
#include <prometheus/counter.h>
#include <prometheus/exposer.h>
#include <prometheus/gauge.h>
#include <prometheus/histogram.h>
#include <prometheus/registry.h>Steps to reproduce:
- Build a DeepStream application with Prometheus metrics enabled using the above headers.
CMakeLists.txt
find_package(prometheus-cpp REQUIRED)
target_link_libraries(${PROJECT_NAME} prometheus-cpp-core prometheus-cpp-pull)
metrics_manager.hpp
// prometheus-cpp headers
#include <prometheus/counter.h>
#include <prometheus/exposer.h>
#include <prometheus/gauge.h>
#include <prometheus/histogram.h>
#include <prometheus/registry.h>
#include <iostream>
#include <thread>
class MetricsManager {
private:
const std::string address_prometheus;
public:
std::atomic<bool> running{true};
prometheus::Gauge *my_gauge = nullptr;
std::shared_ptr<prometheus::Registry> registry;
prometheus::Family<prometheus::Gauge> *gauge_family = nullptr;
std::unique_ptr<prometheus::Exposer> exposer = nullptr;
prometheus::Counter *counter = nullptr;
prometheus::Family<prometheus::Counter> *counter_family = nullptr;
void metrics_loop();
void setup_prometheus();
MetricsManager();
MetricsManager(const std::string &address);
~MetricsManager(); // keep only if you define it
};metrics_manager.cpp
#include "metrics_manager.hpp"
MetricsManager::MetricsManager() {}
MetricsManager::MetricsManager(const std::string& address)
: address_prometheus(address) // <- initialize const member here
{}
void MetricsManager::setup_prometheus() {
// Start the HTTP server on 0.0.0.0:8080
exposer = std::make_unique<prometheus::Exposer>(address_prometheus);
// Set up Prometheus as before...
registry = std::make_shared<prometheus::Registry>();
gauge_family = &prometheus::BuildGauge()
.Name("frame_delay")
.Help("Delay between frames")
.Register(*registry);
my_gauge = &gauge_family->Add({{"source", "camera1"}});
counter_family = &prometheus::BuildCounter()
.Name("frames_received_total")
.Help("Total frames received")
.Register(*registry);
counter = &counter_family->Add({{"label", "value"}});
exposer->RegisterCollectable(registry);
}
void MetricsManager::metrics_loop() { // prometheus::Gauge *my_gauge
while (running) {
// std::cout << "metrics_loop" << std::endl;
counter->Increment();
// simulate updating a metric
my_gauge->Set(static_cast<double>(rand() % 100));
std::this_thread::sleep_for(std::chrono::seconds(2));
}
}
MetricsManager::~MetricsManager() {
std::cout << "MetricsManager destroyed" << std::endl;
}- Run the my DeepStream program
- Send a REST request to add new stream to Deepstream pipeline using
curl.
curl -v -XPOST 'http://localhost:9456/api/v1/stream/add' -d '{
"key": "sensor",
"event": {
"camera_id": "uniqueSensorID1",
"camera_name": "front_door",
"camera_url": "file:///root/P.mp4",
"change": "camera_streaming",
"metadata": {
"resolution": "1920 x1080",
"codec": "h264",
"framerate": 30
}
},
"headers": {
"source": "vst",
"created_at": "2021-06-01T14:34:13.417Z"
}
}'- Observe that the server crashes with the
basic_string::_M_construct null not validerror. gdb log erorr:
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
Thread 24 "civetweb-worker" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff677fe000 (LWP 45804)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140734929821696) at ./nptl/pthread_kill.c:44
44 ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt full
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140734929821696) at ./nptl/pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = 0x7fff677fe000
old_mask = {__val = {0, 0, 0, 0, 2314898798224089137, 3762229861064646688, 7289076169237933879, 2968470289586871081, 0, 0, 0, 0, 2333274052185060197, 1835057137129123700, 9187138739565169161, 9114541635029792639}}
ret = <optimized out>
#1 __pthread_kill_internal (signo=6, threadid=140734929821696) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140734929821696, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff73ed476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#4 0x00007ffff73d37f3 in __GI_abort () at ./stdlib/abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x7ffff75c6860 <stderr>}
sigs = {__val = {32, 0 <repeats 15 times>}}
#5 0x00007ffff7676b9e in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff768220c in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007ffff7682277 in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007ffff76824d8 in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007ffff7679344 in std::__throw_logic_error(char const*) () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007ffff409762e in RequestHandler::handle(CivetServer*, mg_connection*) () at ///opt/nvidia/deepstream/deepstream-7.1/lib/libnvds_rest_server.so
#11 0x00007ffff7bca0e3 in CivetServer::requestHandler(mg_connection*, void*) () at /usr/local/lib/libprometheus-cpp-pull.so.1.3
#12 0x00007ffff7bda7d9 in handle_request () at /usr/local/lib/libprometheus-cpp-pull.so.1.3
#13 0x00007ffff7bdbe52 in worker_thread () at /usr/local/lib/libprometheus-cpp-pull.so.1.3
#14 0x00007ffff743fac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
ret = <optimized out>
pd = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737488340464, 5161855359760523453, 140734929821696, 0, 140737341814736, 140737488340816, -5162155524108001091, -5161839176148393795}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#15 0x00007ffff74d1850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
curl log :
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 127.0.0.1:9456...
* Connected to localhost (127.0.0.1) port 9456 (#0)
> POST /api/v1/stream/add HTTP/1.1
> Host: localhost:9456
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Length: 367
> Content-Type: application/x-www-form-urlencoded
>
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from serverExpected behavior:
DeepStream REST server should run normally with Prometheus metrics enabled.
Actual behavior:
Server crashes with a std::logic_error whenever Prometheus is enabled.
Notes:
- Disabling Prometheus in the project completely removes the issue.
- This looks like a runtime conflict, possibly related to REST/HTTP handling or symbol collisions.