Skip to content

USAGE_GUIDE.md

Latest commit

 

History

History
332 lines (243 loc) · 8.64 KB

USAGE_GUIDE.md

File metadata and controls

332 lines (243 loc) · 8.64 KB
Raw

RtspGStreamerLib - Usage Guide

A comprehensive guide for using RtspGStreamerLib to capture RTSP video frames in .NET applications.

Overview

RtspGStreamerLib provides a simple API to capture frames from RTSP video streams using GStreamer. It uses direct P/Invoke calls to the native GStreamer libraries, eliminating the need for any NuGet dependencies.

Features

  • Direct P/Invoke - No NuGet dependencies, uses system-installed GStreamer
  • Frame callbacks - Process each frame in memory as it arrives
  • Hardware acceleration - Automatic HW decoding support on ARM64 devices
  • H.265/HEVC support - Full codec support (H.264 also available)
  • BGR format - Output ready for OpenCV and computer vision processing
  • Multi-platform - Windows x64, Linux x64, Linux ARM64

Installation

1. Install GStreamer

Linux ARM64 (Raspberry Pi, Khadas VIM3, Jetson)

sudo apt update
sudo apt install -y libgstreamer1.0-0 libgstreamer1.0-dev \
                    gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
                    gstreamer1.0-plugins-bad gstreamer1.0-libav \
                    gstreamer1.0-tools gstreamer1.0-omx

Linux x64

sudo apt update
sudo apt install -y libgstreamer1.0-0 libgstreamer1.0-dev \
                    gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
                    gstreamer1.0-plugins-bad gstreamer1.0-libav

Windows x64

  1. Download from: https://gstreamer.freedesktop.org/download/
  2. Install both Runtime + Development (MSVC x86_64)
  3. Add to PATH: C:\Program Files\gstreamer\1.0\msvc_x86_64\bin
  4. Restart your terminal/IDE

2. Verify Installation

# Linux
ldconfig -p | grep gstreamer
# Should show: libgstreamer-1.0.so.0

# Windows
where gst-launch-1.0

Basic Usage

Simple Example

using RtspGStreamerLib;

class Program
{
    static void Main(string[] args)
    {
        // Initialize GStreamer (call once at startup)
        RtspFrameCapture.Initialize();

        // Create capture instance
        using var capture = new RtspFrameCapture();

        // Frame received callback
        capture.OnFrameReceived += (frame) =>
        {
            Console.WriteLine($"Frame: {frame.Width}x{frame.Height}");
            Console.WriteLine($"Format: {frame.Format}");
            Console.WriteLine($"Data size: {frame.Data.Length} bytes");
            Console.WriteLine($"Stream timestamp: {frame.StreamTimestamp}");

            // Process frame.Data here...
        };

        // Error callback
        capture.OnError += (error) =>
        {
            Console.WriteLine($"Error: {error}");
        };

        // Start capture
        string rtspUrl = "rtsp://user:password@192.168.1.100:554/stream";
        bool useHardwareAccel = false; // Set true on ARM64

        if (capture.Start(rtspUrl, useHardwareAccel))
        {
            Console.WriteLine("Capture started! Press any key to stop...");
            Console.ReadKey();
            capture.Stop();
        }
    }
}

Running the Example

cd RtspGStreamerExample

# Software decoding (all platforms)
dotnet run "rtsp://user:password@192.168.1.100:554/stream"

# Hardware decoding (ARM64 only)
dotnet run "rtsp://user:password@192.168.1.100:554/stream" 1

Frame Data Format

Frames are delivered in BGR format (3 bytes per pixel):

Memory layout: [B0, G0, R0, B1, G1, R1, B2, G2, R2, ...]
Total size: Width x Height x 3 bytes

Accessing Pixels

// Access pixel at position (x, y)
int index = (y * frame.Width + x) * 3;
byte blue  = frame.Data[index];
byte green = frame.Data[index + 1];
byte red   = frame.Data[index + 2];

Converting to Other Formats

// Convert BGR to RGB (if needed)
for (int i = 0; i < frame.Data.Length; i += 3)
{
    byte temp = frame.Data[i];        // Blue
    frame.Data[i] = frame.Data[i + 2]; // Red -> Blue position
    frame.Data[i + 2] = temp;          // Blue -> Red position
}

Timestamps

The library provides two types of timestamps for each frame:

StreamTimestamp (Recommended)

The PTS (Presentation Timestamp) from the RTSP stream. This is the timestamp assigned by the camera with nanosecond precision.

// Get precise time between frames
double deltaSeconds = (currentFrame.StreamTimestamp - previousFrame.StreamTimestamp).TotalSeconds;

ReceivedAt

The DateTime when the frame was received in your C# code. This includes network latency, decoding time, and thread scheduling delays.

// Only use for logging/debugging
Console.WriteLine($"Frame received at: {frame.ReceivedAt}");

See TIMESTAMPS.md for detailed information about timestamp precision.

Advanced Configuration

Adjusting Latency

Modify the latency parameter in RtspFrameCapture.cs:

// Lower latency (50-100ms): Faster response, potentially less stable
"rtspsrc location=\"{rtspUrl}\" protocols=tcp latency=50 ! "

// Higher latency (500ms+): More buffering, more stable
"rtspsrc location=\"{rtspUrl}\" protocols=tcp latency=500 ! "

Changing Output Format

// BGR (default, 3 bytes/pixel) - OpenCV compatible
"videoconvert ! video/x-raw,format=BGR ! "

// RGBA (4 bytes/pixel)
"videoconvert ! video/x-raw,format=RGBA ! "

// Grayscale (1 byte/pixel)
"videoconvert ! video/x-raw,format=GRAY8 ! "

Limiting Frame Rate

Add a frame rate limiter to reduce CPU usage:

// Process only 10 FPS
"videoconvert ! videorate ! video/x-raw,framerate=10/1,format=BGR ! "

Using H.264 Streams

By default, the library uses H.265. For H.264 streams:

string pipeline =
    $"rtspsrc location=\"{rtspUrl}\" protocols=tcp latency=200 ! " +
    "rtph264depay ! h264parse ! " +
    $"{(useHardwareAccel ? "v4l2h264dec" : "avdec_h264")} ! " +
    "videoconvert ! video/x-raw,format=BGR ! " +
    "appsink name=sink emit-signals=false max-buffers=1 drop=true";

Performance

ARM64 with Hardware Acceleration (Khadas VIM3)

Resolution FPS CPU Usage RAM Latency
2560x1440 30 ~10-15% ~60 MB ~200ms
1920x1080 30 ~8-12% ~40 MB ~150ms

x64 with Software Decoding

Resolution FPS CPU Usage RAM Latency
2560x1440 30 ~25-30% ~100 MB ~250ms
1920x1080 30 ~15-20% ~70 MB ~200ms

Troubleshooting

DLL/Library Not Found

Linux:

# Check if library is installed
ldconfig -p | grep gstreamer

# Install if missing
sudo apt install libgstreamer1.0-0

Windows:

# Check if in PATH
where gst-launch-1.0

# If not found, add GStreamer bin folder to PATH

No Frames Received

  1. Test RTSP URL with gst-launch:
gst-launch-1.0 rtspsrc location="rtsp://your-url" ! fakesink

  1. Check codec - ensure it matches (H.264 vs H.265)

  2. Try switching protocols:

// TCP (default, more reliable)
protocols=tcp

// UDP (lower latency, may have packet loss)
protocols=udp

High CPU on ARM64

  1. Verify hardware decoder is available:
gst-inspect-1.0 v4l2h265dec
  1. Enable hardware acceleration:
capture.Start(rtspUrl, useHardwareAccel: true);

Thread Safety Notes

  • Frame callbacks execute on a background thread (not the main thread)
  • Use locks when accessing shared data from callbacks
  • Frame data is copied to managed memory (safe to use after callback returns)
  • Always call Dispose() or use using to release GStreamer resources

Architecture

┌─────────────────┐
│   Your App C#   │
│  (Processing)   │
└────────┬────────┘
         │ OnFrameReceived(VideoFrame)
         │
┌────────▼────────┐
│RtspFrameCapture │
│   (C# Wrapper)  │
└────────┬────────┘
         │ P/Invoke
         │
┌────────▼────────┐
│   GStreamer     │
│  (Native C/C++) │
└────────┬────────┘
         │ RTSP/RTP
         │
┌────────▼────────┐
│   IP Camera     │
│  (H.265 Stream) │
└─────────────────┘

Important Notes

  1. Thread safety: Callbacks run on a background thread. Use locks for shared data.

  2. Memory: Frames are copied to managed memory. Process and release quickly for high throughput.

  3. BGR format: GStreamer outputs BGR (not RGB). Remember this when integrating with libraries that expect RGB.

  4. Disposable: Always use using or call Dispose() to release GStreamer resources.