Omniseer

Omniseer is an edge-to-cloud embodied AI project built around open-vocabulary perception on a ROCK 5B+ mobile robot. The current system captures camera frames, runs YOLO-World inference on the Rockchip NPU, publishes typed ROS 2 detections and performance telemetry, and exposes optional operator diagnostics over gRPC and SRT.

The active portfolio deliverable is a reproducible perception evaluation loop:

camera -> RKNN YOLO-World -> ROS detections + telemetry -> experiment bundle
                                                           -> laptop/cloud review

Robot-side inference, ROS publication, telemetry, simulation, firmware, gateway control, and preview streaming are implemented. Runtime class updates in the native RKNN bridge, structured experiment recording, cloud synchronization, and a hosted review dashboard are planned next. Autonomous object search and capture are not part of the active deliverable.

Current Capabilities

• Hardware-accelerated V4L2 -> RGA -> RKNN producer/consumer vision pipeline.
• YOLO-World text-embedding preparation and bounded detection post-processing.
• Typed /yolo/detections and /vision/perf ROS 2 contracts.
• JSONL stage telemetry, rolling performance summaries, and offline analysis tools.
• ROS 2 simulation and real-hardware bringup with firmware and micro-ROS integration.
• gRPC system status and preview control with on-demand SRT video export.
• Six-lane GitHub CI covering lint, ROS, Gazebo smoke, portable vision, firmware, and docs.

Documentation

• Project documentation
• System architecture
• Edge-to-cloud perception
• CI/CD overview

Common Commands

The root scripts/omni entrypoint is the supported front door for common local workflows:

scripts/omni build ros
scripts/omni test ros
scripts/omni run sim
scripts/omni run real --phase 0.5
scripts/omni check real-perception
scripts/omni flash teensy
scripts/omni docs build

For headless Teensy 4.1 flashing in Docker or over SSH, scripts/omni flash teensy wraps the existing firmware helper.

Current Boundary

GitHub CI validates portable software and simulation contracts. Camera, RGA, RKNN, NPU, sensor, motor, and firmware-flash behavior still require validation on the target hardware. See the CI/CD documentation for the exact coverage boundary.