Skip to content

CLAUDE.md

Latest commit

 

History

History
289 lines (212 loc) · 8.79 KB

CLAUDE.md

File metadata and controls

289 lines (212 loc) · 8.79 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

RtspGStreamerLib is a .NET library for capturing RTSP video frames using GStreamer with direct P/Invoke bindings. Designed for embedded devices and real-time video processing applications.

Key features:

  • Zero NuGet dependencies - Direct P/Invoke to native GStreamer libraries
  • Hardware acceleration - Automatic HW decoding on ARM64 devices (Khadas VIM3, Raspberry Pi, Jetson)
  • H.265/HEVC support - Full codec support via GStreamer
  • Precise timestamps - Uses RTSP stream PTS (Presentation Timestamp) for accurate timing
  • Multi-platform - Windows x64, Linux x64, Linux ARM64
  • Target framework - .NET 8.0

Solution Structure

The solution (RtspGStreamerLib.slnx) contains 4 projects:

RtspGStreamerLib - Main library (for NuGet distribution)

  • Location: RtspGStreamerLib/
  • Core classes:
  • Architecture: Uses appsink element to pull frames from GStreamer pipeline into managed memory

RtspGStreamerExample - Usage example with library reference

  • Location: RtspGStreamerExample/
  • Demonstrates: Connecting to RTSP stream, receiving frames, FPS calculation, timestamp handling
  • Usage: dotnet run <rtsp-url> [hw-accel] [save-interval]

RtspGStreamerInterop - Standalone example (same functionality, no library reference)

Build Commands

# Build entire solution
dotnet build RtspGStreamerLib.slnx

# Build specific configuration
dotnet build -c Release

# Build individual project
dotnet build RtspGStreamerLib/RtspGStreamerLib.csproj

# Create NuGet package
cd RtspGStreamerLib
dotnet pack -c Release
# Package created at: bin/Release/RtspGStreamerLib.1.0.x.nupkg

Running Examples

RtspGStreamerExample (Recommended)

cd RtspGStreamerExample

# Software decoding (works on all platforms)
dotnet run "rtsp://user:password@192.168.0.124:554/stream"

# Hardware decoding (ARM64 only - uses v4l2h265dec)
dotnet run "rtsp://user:password@192.168.0.124:554/stream" 1

# With frame save interval (seconds)
dotnet run "rtsp://user:password@192.168.0.124:554/stream" 0 5

System Dependencies

Linux ARM64 (Khadas VIM3, Raspberry Pi, Jetson)

sudo apt install -y libgstreamer1.0-0 libgstreamer1.0-dev \
                    gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
                    gstreamer1.0-plugins-bad gstreamer1.0-libav \
                    gstreamer1.0-omx gstreamer1.0-tools

Linux x64

sudo apt install -y libgstreamer1.0-0 libgstreamer1.0-dev \
                    gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
                    gstreamer1.0-plugins-bad gstreamer1.0-libav \
                    gstreamer1.0-tools

Windows x64

Architecture and Design Patterns

GStreamer Pipeline Flow

RTSP Stream → rtspsrc → rtph265depay → h265parse → decoder → videoconvert → appsink → C# callback
                                                      ↓
                                          avdec_h265 (software)
                                          v4l2h265dec (ARM64 HW)

Frame Capture Pipeline (RtspFrameCapture.cs)

  1. Initialize GStreamer: RtspFrameCapture.Initialize() calls gst_init()
  2. Create pipeline: Build GStreamer pipeline string with appropriate decoder
  3. Start capture thread: Background thread pulls samples from appsink
  4. Process samples:
    • Extract PTS (Presentation Timestamp) from buffer
    • Map buffer to managed memory
    • Copy frame data to byte array
    • Create VideoFrame with timestamp and metadata
    • Invoke callback on user code

Timestamp Handling

The library provides precise stream timestamps via frame.StreamTimestamp:

capture.OnFrameReceived += (frame) =>
{
    // StreamTimestamp: Real RTSP PTS from camera (nanosecond precision)
    Console.WriteLine($"Stream time: {frame.StreamTimestamp}");

    // ReceivedAt: C# reception time (includes network delay)
    Console.WriteLine($"Received at: {frame.ReceivedAt}");
};

See docs/TIMESTAMPS.md for detailed explanation.

P/Invoke Design

All native GStreamer calls are centralized in RtspGStreamerLib/GStreamerNative.cs:

  • Core GStreamer (gstreamer-1.0-0.dll/.so): Init, pipeline parsing, state management
  • AppSink (gstapp-1.0-0.dll/.so): Pull samples from pipeline
  • Buffer operations: Memory mapping, timestamp extraction, caps parsing

Note: gst_buffer_get_pts and gst_buffer_get_dts are macros in GStreamer, not exported functions. The library reads PTS/DTS directly from the GstBuffer structure.

Platform-specific library loading handled automatically by .NET runtime.

Hardware Acceleration

Decoder selection in RtspFrameCapture.cs:

string decoder = useHardwareAccel ? "v4l2h265dec" : "avdec_h265";
  • v4l2h265dec: Hardware decoder for ARM64 (10-15% CPU @ 1440p)
  • avdec_h265: Software decoder (25-30% CPU on x64, 60%+ on ARM64)

Key Configuration Points

Pipeline Tuning

Latency (RtspFrameCapture.cs):

"rtspsrc location=\"{rtspUrl}\" protocols=tcp latency=200 ! "
  • Lower latency (50-100ms): Faster response, more CPU, less stable
  • Higher latency (500ms): More buffering, stable, higher delay

Frame dropping (RtspFrameCapture.cs):

"appsink name=sink emit-signals=false max-buffers=1 drop=true"
  • max-buffers=1 drop=true: Always get latest frame, discard old ones

Output format (RtspFrameCapture.cs):

"videoconvert ! video/x-raw,format=BGR ! "
  • BGR: 3 bytes/pixel, OpenCV-compatible
  • Alternatives: RGBA (4 bytes), GRAY8 (1 byte for grayscale)

Adding Frame Rate Limiting

Insert after videoconvert to reduce CPU:

"videoconvert ! videorate ! video/x-raw,framerate=10/1,format=BGR ! "

Changing Codecs

H.264 streams (instead of H.265):

"rtspsrc location=\"{rtspUrl}\" protocols=tcp latency=200 ! " +
"rtph264depay ! h264parse ! " +
(useHardwareAccel ? "v4l2h264dec" : "avdec_h264") + " ! " +
"videoconvert ! video/x-raw,format=BGR ! " +
"appsink name=sink emit-signals=false max-buffers=1 drop=true"

Documentation

Documentation in docs/:

Troubleshooting

DLL/SO Not Found

Linux:

# Check if library is installed
ldconfig -p | grep gstreamer
# Should show: libgstreamer-1.0.so.0

Windows:

# Check if in PATH
where gst-launch-1.0

No Frames Received

  1. Test RTSP URL manually:
gst-launch-1.0 rtspsrc location="rtsp://your-camera" ! fakesink

  1. Check codec - ensure camera uses H.265 or H.264

  2. Try UDP if TCP fails (modify protocols=tcp → protocols=udp)

High CPU on ARM64

Verify hardware decoder is available:

gst-inspect-1.0 v4l2h265dec
# Should show plugin info, not "No such element"

Enable hardware acceleration in code:

capture.Start(rtspUrl, useHardwareAccel: true);

Performance Benchmarks

Khadas VIM3 (ARM64 with HW acceleration)

  • Stream: 2560x1440 @ 30fps
  • CPU: 10-15% (v4l2h265dec)
  • RAM: 60 MB
  • Latency: ~200ms

Windows x64 (Software decoding)

  • Stream: 2560x1440 @ 30fps
  • CPU: 25-30% (avdec_h265)
  • RAM: 100 MB
  • Latency: ~250ms

Common Development Tasks

# Clean build artifacts
dotnet clean

# Restore packages
dotnet restore

# Run with debug logging
GST_DEBUG=3 dotnet run

# Check GStreamer elements
gst-inspect-1.0 v4l2h265dec
gst-inspect-1.0 avdec_h265

# Test pipeline manually
gst-launch-1.0 rtspsrc location="rtsp://camera" ! rtph265depay ! h265parse ! avdec_h265 ! autovideosink

Thread Safety Notes

  • Frame callbacks execute on background thread (not main thread)
  • Use locks when accessing shared data from callbacks
  • Frame data is copied to managed memory (safe to use after callback returns)
  • Always call Dispose() or use using to release GStreamer resources