Skip to content

Conversation

ryanwilson
Copy link
Member

This change introduces video streaming capabilities to the Firebase AI Logic library, mirroring the existing audio streaming API.

Key features:

  • A new VideoHelper class to manage camera input and frame capture.
  • startVideoConversation and stopVideoConversation methods in LiveSession to control the video stream.
  • Image frames are captured, scaled to a maximum dimension of 2048 pixels, and encoded as JPEGs before being sent to the backend.
  • Image processing is offloaded to a background coroutine to avoid blocking the main thread and ensure smooth video streaming.
  • Camera resolution selection is optimized to choose the largest available resolution for the best quality.
  • Image scaling is performed efficiently using inSampleSize to minimize memory and CPU usage.

This change introduces video streaming capabilities to the Firebase AI Logic library, mirroring the existing audio streaming API.

Key features:
- A new `VideoHelper` class to manage camera input and frame capture.
- `startVideoConversation` and `stopVideoConversation` methods in `LiveSession` to control the video stream.
- Image frames are captured, scaled to a maximum dimension of 2048 pixels, and encoded as JPEGs before being sent to the backend.
- Image processing is offloaded to a background coroutine to avoid blocking the main thread and ensure smooth video streaming.
- Camera resolution selection is optimized to choose the largest available resolution for the best quality.
- Image scaling is performed efficiently using `inSampleSize` to minimize memory and CPU usage.

Note: The tests for this feature could not be run due to a persistent issue with locating the Android SDK in the testing environment.
Copy link
Contributor

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

@google-oss-bot
Copy link
Contributor

1 Warning
⚠️ Did you forget to add a changelog entry? (Add the 'no-changelog' label to the PR to silence this warning.)

Generated by 🚫 Danger

@google-oss-bot
Copy link
Contributor

@google-oss-bot
Copy link
Contributor

The public api surface has changed for the subproject firebase-ai:
error: Added method com.google.firebase.ai.type.LiveSession.isVideoConversationActive() [AddedMethod]
error: Added method com.google.firebase.ai.type.LiveSession.startVideoConversation(String,kotlin.coroutines.Continuation<? super kotlin.Unit>) [AddedMethod]
error: Added method com.google.firebase.ai.type.LiveSession.stopVideoConversation() [AddedMethod]

Please update the api.txt files for the subprojects being affected by this change by running ./gradlew ${subproject}:generateApiTxtFile. Also perform a major/minor bump accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants