Release 4.0.0

4.0.0 (2024-04-22)

VisionCamera V4 is finally here! 🥳🥳🥳🥳🚀💪

The most anticipated features are:

• Skia Frame Processors: Finally it's reality - draw directly onto a Frame using react-native-skia! It has never been simpler to draw something onto a Frame, with a native iOS/Android app you'd spend days if not weeks on this.

  const frameProcessor = useSkiaFrameProcessor((frame) => {
    'worklet'
  
    // draw frame itself
    frame.render()
  
    // draw red rectangle into center of frame
    const centerX = frame.width / 2
    const centerY = frame.height / 2
    const rect = Skia.XYWHRect(centerX, centerY, 150, 150)
    const paint = Skia.Paint()
    paint.setColor(Skia.Color('red'))
    frame.drawRect(rect, paint)
  }, [])

• CameraX rewrite: muuuuch more stability!!
• GPS Location Tags: Adds location tags to photos and videos.

If you followed along the VisionCamera V2 -> V3 journey, you might remember that I switched from the (at-that-time-) immature CameraX library, to the low-level almost C-like Camera2 API.
The motivation behind that was that CameraX was too limiting at that time, and I wanted to gain full control over the hardware sensor and output streams.
While this worked great on my test devices, it seems like Camera2 wasn't implemented the same on all devices, and countless of weird issues arised, such as Samsung phones completely crashing, photos being rotated, frame processors running really slow on Huawei, some non-null Camera properties actually being null, etc. - the list goes on.
While I was fighting my way through Android's low-level Camera2 API, CameraX started gaining more and more interesting features, and just recently they built a new API: stream sharing.
With stream sharing, CameraX now supports all the features that VisionCamera has, and so I decided to fully rewrite the Android codebase to CameraX.
I was pretty impressed with what I built in just a few months using Camera2 APIs, but in reality this is becoming unmaintainable as countless of phone manufacturers just don't really care about Camera2 APIs, and I can't be the person monkeypatching that all the time.
Compared to VisionCamera V3, VisionCamera V4 is now much more stable, and brings a ton of new features.

✨ Features

• Fully rewrite the Android codebase to CameraX. This brings huge stability improvements overall, and should fix countless of bugs 🥳
• Add Skia integration to provide drawable Frame Processors! 🥳🎨
  • Add useSkiaFrameProcessor hook to create a drawable Frame Processor
  • Inside a useSkiaFrameProcessor the caller can directly draw to the frame as it extends a SkCanvas
  • Use all Skia.* APIs
  • Render the Frame with optional shaders using frame.render(shader)
• Add GPS/EXIF Location Tags to captured photos or videos 📍
  • Add useLocationPermission hook to get location permission
  • Add enableLocation prop to <Camera>
  • Add $VCEnableLocation config option to Podfile to disable location APIs (e.g. if Apple Review is not happy with unused location APIs in your code)
• Add photoQualityBalance prop to <Camera> to optimize the Camera pipeline for the given prioritization (speed, balanced (default) or quality)
• Add preview prop to <Camera> to enable or disable the preview view/stream.
• Add androidPreviewViewType prop to <Camera> to specify which preview implementation to use (surface-view is faster and supports HDR, texture-view supports transparency, masks, rotations and clipping)
• New takeSnapshot(..) API to take instant snapshots of the video/preview streams. This can be used for intermediate results before takePhoto(..) completes. 📸
• Use stream sharing to drop limitation of concurrent outputs - now photo, video, frameProcessor, preview and codeScanner can all be enabled at the same time!
• Add photo HDR support to Android (using vendor-specific extensions) 🥳
• Add video HDR support to Android (using 10-bit video HDR) 🥳
• Always support yuv and rgb pixel-formats
  • yuv is more efficient (uses ~60% less memory than rgb, and does not need conversion)
  • rgb requires explicit conversion from yuv and will increase latency
• Rewrite FPS Graph in JS to make it look more sexy and simplify native build (less native UI code)
• Add support for legacy Camera1 devices
• Lower minSdk level from 23 to 21.
• Separate VisionCamera into three submodules/subspecs:
  • VisionCamera/Core: The core library which could be used in a native iOS/Android app
  • VisionCamera/React: React Native bindings for VisionCamera/Core
  • VisionCamera/FrameProcessors: React Native bindings for synchronous onFrame callbacks using react-native-worklets-core

❌ Deprecations

• Remove qualityPrioritizaion parameter in takePhoto(..) in favor of new photoQualityBalance prop on <Camera>
• Remove enablePrecapture parameter in takePhoto(..) because CameraX now properly runs precapture sequences with device-quirks ironed out
• Remove native pixel-format since yuv is native
• Remove enableGpuBuffers since CameraX now managed such optimizations

💨 Performance Improvements

• Optimize for concurrent configure { ... } abort calls and give higher priority to newer update
• Improve performance for some JNI calls (SharedArray and FrameProcessor::call)

🐛 Bug Fixes

• Fully rewrite the Android codebase to CameraX. This brings huge stability improvements overall, and should fix countless of bugs.
• Fix onStarted/onStopped sometimes being called a bit too early
• Fix shutter sound playing when phone is in silent mode
• Fix Camera not starting and appearing black on some devices
• Fix Camera crashing because no resolution could be found
• Fix Camera crashing the entire Android OS because GPU buffers were enabled
• Automatically fall-back to a supported videoCodec if the given video-codec is not supported (e.g. h265 -> h264)
• Fix Camera flipping not working sometimes

• Use correct UIManager in findCameraView depending on the architecture (#2702) (5223f5b)

📚 Documentation

• Update docs for Skia Frame Processors
• Update docs for takeSnapshot
• Update docs for V4
• Add a ton of new Frame Processor plugins by @gev2002 (#2698) (f396ea7)
• Update FRAME_PROCESSOR_PLUGINS.mdx (#2705) (04f89c4)

Check out the VisionCamera V4 PR (#2623) for a full list of changes.

Huge shoutout to @wcandillon for helping me get the required changes into react-native-skia! We spent a lot of time debugging weird GPU issues and optimizing the code.
These are the Skia PRs needed for the VisionCamera V4 Skia Frame Processors feature:

• Shopify/react-native-skia#1365
• Shopify/react-native-skia#1377
• Shopify/react-native-skia#1539
• Shopify/react-native-skia#2261
• Shopify/react-native-skia#2344
• Shopify/react-native-skia#2345
• Shopify/react-native-skia#2356
• Shopify/react-native-skia#2357
• Shopify/react-native-skia#2372
• Shopify/react-native-skia#2385
• Shopify/react-native-skia#2388
• Shopify/react-native-skia#2389
• Shopify/react-native-skia#2390

Release 4.0.0-beta.19

4.0.0-beta.19 (2024-04-22)

✨ Features

• Upgrade to CameraX alpha5 (#2771) (5169271)

🐛 Bug Fixes

• Fix delegate guard (884b89d)
• Fix ImageAnalysis is not supported when Extension is enabled error (#2770) (aec14b2)
• Fix Timer is already canceled (c0f3ee0)
• Fix Android import path due to new subfolder (406b741)
• Make FpsSampleCollectorDelegate weak (e93c522)
• Separate FpsSampleCollectorDelegate (83cac76)

📚 Documentation

• Add a ton of new Frame Processor plugins by @gev2002 (#2698) (f396ea7)
• Update FRAME_PROCESSOR_PLUGINS.mdx (#2705) (04f89c4)

Release 4.0.0-beta.18

4.0.0-beta.18 (2024-04-21)

✨ Features

• Rebuild FPS Graph as JS component (#2769) (4163007)

🐛 Bug Fixes

• Fix already bound to a different lifecycle error when flipping camera (#2768) (a1b7602)
• Simplify cpp/ codebase into android/ios folders (#2766) (1f4381a)

Release 4.0.0-beta.17

4.0.0-beta.17 (2024-04-21)

✨ Features

• Separate VisionCamera into React, Core and FrameProcessors subspecs (#2764) (464ea94)
• Separate VisionCamera into two subspecs (#2761) (f7dbb24)

🐛 Bug Fixes

• Fix Promise resolve wrong name (e124c00)
• Fix build for Android because of Worklet usage (9fcb61a)
• Refactor Android codebase into core, react and frameprocessors (#2765) (991dc79)

Release 4.0.0-beta.16

4.0.0-beta.16 (2024-04-19)

✨ Features

• Optionally compile out Location APIs (#2751) (9b901df)

🐛 Bug Fixes

• Also override onProviderDisabled to prevent abstract method crash (b0a5585)
• Clean up Skia resources on Worklet context to avoid race condition (#2754) (b8e44ab)
• Fix minFps being larger in Range than maxFps (#2755) (16ccfdf)
• Make dependencies (ModuleProxy) actually optional (#2750) (c3098db)
• Remove enableGpuBuffers (b751f2d)

📚 Documentation

• Update docs for V4 (#2747) (28db499)

Release 4.0.0-beta.15

4.0.0-beta.15 (2024-04-17)

This is the first beta for Skia Frame Processors! 🥳
You can try this today in your app by using the useSkiaFrameProcessor hook:

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'

  // 1. create blur filter
  const blurRadius = 10
  const blurFilter = Skia.ImageFilter.MakeBlur(blurRadius, blurRadius, TileMode.Repeat, null)

  // 2. wrap blur filter in Paint
  const paint = Skia.Paint()
  paint.setImageFilter(blurFilter)

  // 3. render image with the blur filter paint
  frame.render(paint)
}, [])

Note: Skia Frame Processors require react-native-worklets-core 1.1.1 or higher, react-native-reanimated 3.x.x or higher, and react-native-skia 1.2.1 or higher

Note: You might need to set pixelFormat="rgb", as "yuv" buffers are still work in progress (see Shopify/react-native-skia#2357)

✨ Features

• Skia for V4! 🥳 (#2727) (be9b4d3)

🐛 Bug Fixes

• Fix ResolutionSelector difference abs (#2746) (1b73b69)
• Rename ReactLogger to VisionLogger (b89df9a)
• Use 1280x720 for QR scanner (ca2995d)

Release 4.0.0-beta.14

4.0.0-beta.14 (2024-04-16)

✨ Features

• Add hardware-cost-too-high error for iOS 16 (56a5c8d)

🐛 Bug Fixes

• Always support rgb and yuv, remove native (cd9b84d)
• Fall-back to H.264 if custom codec is not supported (4595b4f)
• Use ResolutionSelectors with fallback values (adcca77)

Release 4.0.0-beta.13

4.0.0-beta.13 (2024-04-04)

🐛 Bug Fixes

• Remove mustPlayShutterSound() check (a6323cf)

Release 4.0.0-beta.12

4.0.0-beta.12 (2024-04-02)

✨ Features

• Disable shutter sound if in silent (86c0f86)

🐛 Bug Fixes

• Remove enablePrecapture (688acd8)
• Use correct UIManager in findCameraView depending on the architecture (#2702) (5223f5b)

Release 4.0.0-beta.11

4.0.0-beta.11 (2024-03-27)

💨 Performance Improvements

• Add @FastNative flags to critical Frame Processor methods (ff2005b)

🐛 Bug Fixes

• Fix RCTDeviceEventEmitter import for RN 0.71.X (#2692) (8a95414)
• Fix wrong orientation enum being send to ImageCapture (#2683) (4c00ffd)

4.0.0-beta.10 (2024-03-21)

🐛 Bug Fixes

• Fix location permission not resolving on iOS (#2672) (a255c24)

4.0.0-beta.9 (2024-03-21)

✨ Features

• Add GPS Location EXIF tagging for photos and videos (enableLocation={true}) (#2665) (c8033dd)

🐛 Bug Fixes

• Fix CodeScanner not working (c0e3a14)
• Fix focus on Android by using PX instead of DP (#2666) (f794d48)
• Remove HardwareBufferUtils (3f113e8)

4.0.0-beta.8 (2024-03-19)

✨ Features

• Basic read-only Frame Processors (ImageAnalysis Use-Case) (#2664) (ca6518f)

🐛 Bug Fixes

• Properly set maxPhotoDimensions on PhotoOutput (#2659) (7f14f25)

4.0.0-beta.7 (2024-03-18)

✨ Features

• Write proper photo metadata (orientation & isMirrored) (#2660) (babed3c)

🐛 Bug Fixes

• Also dont enable if other extension is enabled (5525f2b)
• Fix 10-bit HDR in combination with HDR Extension (19a7b0c)
• Fix Camera not starting because PreviewView isn't laid out (ad746d5)
• Fix capture session configuration error by manually layouting PreviewView (3d68d74)
• Properly check SDR (211783a)
• Remove focus listeners after resetting (2611625)
• Throw correct errors (5fe7608)
• Update outputs if video stabilization or low-light-boost changes (27a4f85)
• Use CREATED as default lifecycle state (1682bb2)

4.0.0-beta.6 (2024-03-14)

✨ Features

• Catch Objective-C init errors on plugin init (fa21983)

🐛 Bug Fixes

• Catch Objective-C errors in iOS Frame Processor Plugins (c1f274f)
• Check format requirements in outputs (79d4554)
• Fix TimeoutException by using STARTED lifecycle (5283020)
• Fix rotation getting (a9093cf)
• Remove didSessionChangeFromOutside (5119639)
• Remove not-compatible-with-outputs error on Android since CameraX supports StreamSharing (8ad08f3)
• Remove unneeded context (aac2162)
• Use dummy metering factory instead of display-oriented (a57d640)

4.0.0-beta.5 (2024-03-08)

✨ Features

• Add type to onShutter (photo or snapshot) (f721d97)
• Remove enableAutoStabilization prop as this was deprecated in iOS 13 (72c8c1b)

🐛 Bug Fixes

• Allow catching errors in .await() (7ec53f7)
• Allow re-creating if session changed from outside (17feff3)
• Don't play shutter sound unless explicitly enabled (f5c0131)
• Export Snapshot.ts (6be93af)
• Fix calling sendAvailableDevicesChangedEvent before module is initialized (b4a9086)
• Only unbind current outputs, not all (50f47c5)

4.0.0-beta.4 (2024-03-01)

✨ Features

• Add androidPreviewViewType prop (dd56925)
• Add cancelRecording (f51921b)
• Add onShutter event (dd49365)
• Add takeSnapshot() to PreviewView (e17eb50)
• Add CodeScanner pipeline (ImageAnalysis) (08d86d4)
• Allow setting custom format in ImageWriter (d0eb702)
• Give V4 🫴 (dd4c8fb)
• Implement native HardwareBuffer rendering for VideoPipeline (00155c9)
• Properly get orientation and isMirrored in FP (a4a78e9)
• Shutter sound and stabilization (7b46390)
• Use CameraX for device details (#2624) (54997a2)
• Use HDR or NIGHT extensions if available (b9a0434)

🐛 Bug F...