[Feature Request] Direct Camera Frame Access API

[kv-michal]

There is currently no way to access raw camera frames from within a ViroARScene or ViroARSceneNavigator. This makes it impossible to implement use cases that require per-frame image processing — such as custom ML inference (e.g. object detection, pose estimation, OCR), QR/barcode scanning pipelines, frame capture for recording, or vision-based game mechanics — without reaching for a completely separate camera library that conflicts or overlaps with ViroReact's own camera session.

---

Describe the solution you'd like

A callback or hook API that exposes each camera frame (or a throttled subset) as
raw pixel data or a GPU texture reference, available from within the AR scene context.

Option A — callback prop on ViroARScene:

<ViroARScene
  onCameraFrame={(frame: ViroCameraFrame) => {
    // frame.width, frame.height
    // frame.pixelBuffer  — Uint8Array (RGBA or YUV)
    // frame.timestamp    — number (ms)
    // frame.orientation  — 'portrait' | 'landscape'
  }}
  cameraFrameRate={15} // throttle to N fps to control perf overhead
>
  ...
</ViroARScene>

Option B — imperative ref API on ViroARSceneNavigator:

const navRef = useRef<ViroARSceneNavigator>(null)

// capture a single frame on demand
const frame = await navRef.current?.captureFrame()

---

Describe alternatives you've considered

• react-native-vision-camera in parallel — creates two competing camera sessions,
  causes conflicts on iOS (AVCaptureSession) and Android (Camera2), increases
  battery/CPU load, and provides no way to correlate frame data with AR world coordinates.

• ViroARSceneNavigator.takeScreenshot — too slow (saves to disk, not real-time),
  no pixel buffer access, not suitable for per-frame ML inference.

• Custom native module — breaks the abstraction, requires maintaining separate
  native code for iOS and Android, and re-implements what ViroReact already has internally.

---

Use cases this would unlock

• On-device ML inference (Core ML / TFLite) on the live AR camera feed
• Real-time QR / barcode scanning inside an AR scene without a second camera
• Custom visual effects or shaders that react to the real world
• Frame capture / video recording of the raw camera stream
• Computer vision pipelines (edge detection, color tracking, motion detection)

---

react-native-vision-camera v4 exposes a useFrameProcessor hook with a
Worklet-based API that runs frame processing on a dedicated thread — a similar
pattern would work well here and avoid blocking the JS/UI thread. Ideally the
frame callback would run off the main thread with Reanimated Worklet or JSI support.

[mrousavy]

Hey - VisionCamera V5 supports streaming Frames since 4 years now - and V5 now exposes more information on a Frame such as Frame.cameraIntrinsicMatrix - which is a 3x3 matrix with column-major ordering. Its origin is the top-left of the Frame.

K = [ fx   0  cx ]
    [  0  fy  cy ]
    [  0   0   1 ]

• fx, fy: focal length in pixels
• cx, cy: principal point in pixels

Same for Depth, which now supports Depth.cameraCalibrationData.

Would this be useful to somehow open up the possibility to connect react-native-vision-camera and ReactVision/viro, to use react-native-vision-camera for streaming the Frames and viro for building the 3D/AR/VR scene?