Skip to content

refactor: update Live API configuration, transcription support, and i…#1224

Merged
thorwebdev merged 1 commit into
mainfrom
thor/update-live-quickstart-python
May 28, 2026
Merged

refactor: update Live API configuration, transcription support, and i…#1224
thorwebdev merged 1 commit into
mainfrom
thor/update-live-quickstart-python

Conversation

@thorwebdev

Copy link
Copy Markdown
Collaborator

…nput handling for model version gemini-3.1-flash-live-preview

fixes #1108

…nput handling for model version gemini-3.1-flash-live-preview
@thorwebdev thorwebdev requested a review from Giom-V May 8, 2026 20:25
@github-actions github-actions Bot added status:awaiting review PR awaiting review from a maintainer component:quickstarts Issues/PR referencing quickstarts folder labels May 8, 2026
@thorwebdev thorwebdev requested a review from kkorpal May 8, 2026 20:25
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly updates the Get_started_LiveAPI.py quickstart example to leverage the latest gemini-3.1-flash-live-preview model. The changes primarily focus on adapting the client configuration and real-time input/output handling to incorporate new features like audio transcription and a more streamlined API for sending media blobs. This ensures the example remains compatible with the most recent API version and demonstrates enhanced interactive capabilities.

Highlights

  • Live API Configuration Update: The quickstart example has been updated to use the new gemini-3.1-flash-live-preview model, replacing the older gemini-2.5-flash-native-audio-preview-12-2025 model.
  • Transcription Support Enabled: Input and output audio transcription capabilities have been enabled within the LiveSession configuration, allowing for real-time text representation of spoken input and model responses.
  • Streamlined Input Handling: The methods for sending text, audio, and visual inputs have been refactored to align with the updated API, including the removal of base64 encoding for image data and the use of types.Blob for media inputs.
  • API Key Environment Variable Change: The environment variable for the API key has been updated from GEMINI_API_KEY to GOOGLE_API_KEY.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Live API quickstart to use the google-genai SDK (v1.0.0+), involving model and API key updates, enabling audio transcriptions, and refactoring the audio reception logic. The review feedback highlights several necessary corrections for SDK compatibility, such as the proper usage of send_realtime_input and send_client_content. Additionally, improvements were suggested regarding the repository's style guide for model selection, the use of constants for sample rates, and the handling of incremental transcription updates.

Comment thread quickstarts/Get_started_LiveAPI.py
Comment thread quickstarts/Get_started_LiveAPI.py
Comment thread quickstarts/Get_started_LiveAPI.py
Comment thread quickstarts/Get_started_LiveAPI.py
Comment thread quickstarts/Get_started_LiveAPI.py
@thorwebdev thorwebdev merged commit fcaac01 into main May 28, 2026
4 checks passed
@thorwebdev thorwebdev deleted the thor/update-live-quickstart-python branch May 28, 2026 18:44
Giom-V added a commit that referenced this pull request Jun 17, 2026
* Bump uuid and googleapis in /quickstarts/file-api (#1212)

Removes [uuid](https://github.com/uuidjs/uuid). It's no longer used after updating ancestor dependency [googleapis](https://github.com/googleapis/google-api-nodejs-client). These dependencies need to be updated together.


Removes `uuid`

Updates `googleapis` from 134.0.0 to 171.4.0
- [Release notes](https://github.com/googleapis/google-api-nodejs-client/releases)
- [Commits](googleapis/google-api-nodejs-client@googleapis-v134.0.0...googleapis-v171.4.0)

---
updated-dependencies:
- dependency-name: uuid
  dependency-version: 
  dependency-type: indirect
- dependency-name: googleapis
  dependency-version: 171.4.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Flash-Lite 3.1 goes GA (#1221)

* Create README.md for the prompting_techniques folder (#1230)



---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Added new notebook Basic_Code_Review to examples (#689)

---------

Co-authored-by: kkorpal <kkorpal@google.com>
Co-authored-by: Shipra <Shi-pra-19@users.noreply.github.com>

* Add Basic_Code_Review example to README (#1218)

* Add Basic_Code_Review example to README

* Update examples/prompting/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Added example for gemini api used to extract insights from github #923 #887 (#1150)

* added example for gemini api used to extract insights from github


---------

Co-authored-by: kkorpal <kkorpal@google.com>

* Add example: EU AI Act compliance checking with Gemini function calling + Agent Module (#1167)

* Add EU AI Act compliance example with Gemini function calling + Agent Module

* style: use triple-quoted string per review feedback

* fix: correct API schema + formatting per review

- status["ethics"] -> status["ai_compliance"] (v0.9.21 schema rename; was
  the KeyError that blocked the reviewer run)
- Remove unnecessary backslash line-continuations in the compliance_run
  prompt string; indent body; newline before closing """
- Replace static 22-row module table with dynamic discovery pointer
  (/api/status, /api/demo, /llms.txt) so the snapshot never goes stale
- Branding: "ethics modules" -> "AI Compliance modules" in human-facing
  markdown; vertical slug documented as ai-compliance with ethics as
  legacy alias (both still accepted by the API)
- Correct ETH_001 EU AI Act mapping (Art. 9 + 10, not 10 + 13)

* fix: prevent trial key collision on reruns

Follow-up to the schema/formatting fixes. Colab testing surfaced a
runtime bug: the Agent Module API returns a masked preview of an
existing trial key (e.g. `am_trial_841...f660`) if you request a new
one while the old one is still active, and the original notebook
wrapper blindly stored the preview — corrupting _trial_key and
breaking every downstream retrieve_compliance_logic call.

Changes:
- Generate a unique per-run agent_id (uuid suffix) so reruns within
  24 hours do not collide with a still-active trial on the server
- Make get_trial_key idempotent (short-circuit if _trial_key already
  set in this session) so Gemini tool loops cannot re-trigger it
- Detect status=="already_active" in the MCP response and refuse to
  store the masked preview; surface a clear error instead
- Remove get_trial_key from the tools list passed to Gemini. Trial
  bootstrap is a setup step, not a runtime tool — the model should
  only see retrieve_compliance_logic and submit_proof_of_value
- Reframe the cell 27 preamble to explain the setup/runtime split

* fix: pass both token and trial_key to submit_pov MCP tool

* fix: print full chat history so cell 30 shows the complete compliance report

* chore: save executed notebook with cell outputs

* fix: add Colab button + README link for CI compliance

- Add Colab badge to notebook title cell (tfo-notebook-buttons CI check)
- Add notebook entry to examples/README.md table (new-example-links CI check)

* chore: retrigger CI checks

* Fix notebook lint (inclusive_language) + nbfmt format

- Rewords 'Agent-native' to 'Purpose-built for autonomous agents' in the branding cell to clear google::inclusive_language lint flag
- Applies tensorflow_docs.tools.nbfmt formatting pass to clear Notebook format check
- No other content changes; all 21 cell outputs preserved

* Fix colab.name metadata + cell 30 prompt (third-person) to clear nbfmt + second_person checks

---------

Co-authored-by: Guillaume Vernade <Giom-V@users.noreply.github.com>

* add gemini-3.5-flash (#1232)

* New agent notebook (#1233)

* Fixing typos

* Adding links to the agents notebooks (#1234)

* docs: fix typos and capitalization consistency in README (#1240)

* docs: fix typos and capitalization consistency in README

* docs: apply gemini suggestion

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Bump qs from 6.15.1 to 6.15.2 in /quickstarts/file-api (#1238)

Bumps [qs](https://github.com/ljharb/qs) from 6.15.1 to 6.15.2.
- [Changelog](https://github.com/ljharb/qs/blob/main/CHANGELOG.md)
- [Commits](ljharb/qs@v6.15.1...v6.15.2)

---
updated-dependencies:
- dependency-name: qs
  dependency-version: 6.15.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix scale= command in Lyria realtime quickstart (#1236)

The scale= parser compared the raw prompt (which still contains the
'scale=' prefix) against enum names like A_FLAT_MAJOR_F_MINOR, so the
match never succeeded and the command always printed 'Matching enum
not found.'

Strip the 'scale=' prefix before comparing, matching how the sibling
bpm= and top_k= handlers already work.

* update live api ws cookbook (#1223)

* Interactions steps (#1237)

* Merge origin/main into interaction-steps and resolve conflict

* steps

* Add environment support

* work

* cleanup

* format

* cleanup

* Add back execution_count

* Standardize Colab buttons and fix URLs across edited notebooks

* feat: update visual model names to GA and add video-to-image example (#1244)

* feat: update visual model names to GA and add video-to-image example

- Updated gemini-3-pro-image-preview to gemini-3-pro-image (Nano Banana Pro GA)
- Updated gemini-3.1-flash-image-preview to gemini-3.1-flash-image (Nano Banana 2 GA)
- Added a new video-to-image example in Get_Started_Nano_Banana.ipynb
- Formatted modified notebooks with nbfmt

* feat(batch): add gemini-3.5-flash to model selector

* style: format banana notebook to clean up PR diff

* style: trim thought signature in Get_Started_Nano_Banana notebook

To reduce file size and prevent git/diff hanging on massive cryptographically signed binary thoughts.

* refactor: update Live API configuration, transcription support, and input handling for model version gemini-3.1-flash-live-preview (#1224)

* Update Get_started_LiveAPI (#1222)

* Update Get_started_LiveAPI

* remove interactivity.

* session resumption update.

* run.

* nbfmt

* Resizing the NB notebook (#1245)

* feat: add example for custom FPS in Grounding notebook (#1243)

* feat: add example for custom FPS in Grounding notebook

* Format custom frame rate explanation in Grounding.ipynb

* Fix formatting of YouTube link and response generation

* Fix formatting issues in Grounding.ipynb

* Fix YouTube link formatting in Grounding.ipynb

* docs: fix typographical and spelling errors in styleguide and quickstarts (#1253)

* docs: fix typographical and spelling errors in styleguide.md

* docs: fix typo in Image_out notebook header

* fix: remove leftover google-generativeai imports from two notebooks (#1246)

Two notebooks had leftover `import google.generativeai as genai` lines
from a prior partial migration. Both notebooks otherwise use the new
`google-genai` SDK correctly via `from google import genai`, and
neither notebook's `%pip install` cell installs `google-generativeai`.
On a clean Colab kernel the dead imports raise `ModuleNotFoundError`;
on a kernel where the legacy package happens to be pre-installed, the
dead imports silently shadow the working `genai` namespace.

- quickstarts/Spatial_understanding.ipynb cell 19: delete the dead
  `import google.generativeai as genai` line; the rest of the cell
  (PIL.Image, io, os, requests, BytesIO) is genuinely used downstream
- examples/Animated_Story_Video_Generation_gemini.ipynb cell 7:
  delete the cell entirely. The cell holds only the dead import

No behavioural change beyond removing dead code that would otherwise
fail or shadow at notebook runtime.

* fix(examples): replace deprecated Live API session.send in Animated Story notebook (#1249)

Use session.send_client_content(turns=..., turn_complete=True) instead of the
deprecated session.send(input=..., end_of_turn=True), which emits a
DeprecationWarning in google-genai (removal not before Q3 2025).

Co-authored-by: Sri Ujwal <sbeeredd04@users.noreply.github.com>

* fix(quickstarts): correct case-sensitive links to Search_Grounding/Code_Execution (#1248)

Three cross-notebook links used lowercase filenames (Search_grounding.ipynb,
Code_execution.ipynb) that 404 on case-sensitive file systems like GitHub. Point
them at the actual files: Search_Grounding.ipynb and Code_Execution.ipynb.

Co-authored-by: Sri Ujwal <sbeeredd04@users.noreply.github.com>

* fix(quickstarts): correct broken Contributing guide link in Template.ipynb (#1247)

The template linked the contributing guide as `Contributing.mg`, which is both a
typo (`.mg`) and a path that does not exist next to the notebook. Point it at the
repo's CONTRIBUTING.md using the same absolute GitHub URL the notebook already
uses elsewhere.

Co-authored-by: Sri Ujwal <sbeeredd04@users.noreply.github.com>

* fix: align error handling notebook with google-genai retries (#1229)

* docs: fix broken/incomplete links and remove duplicate entry in quickstarts README (#1257)

* docs: fix broken/incomplete links and remove duplicate entry in quickstarts README

* Update quickstarts/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* feat: add Live Translation quickstart tutorial and examples (#1259)

* feat: add Live Translation quickstart tutorial and examples

* chore: clean up notebook metadata and reorder cell output attributes

* refactor: Live Translation notebook and script from cookbook (#1261)

* Prohibit yt-dlp usage for YouTube content (#1264)

Added a restriction against using yt-dlp for downloading or streaming YouTube content due to terms of service violations.

* Sketch2Paint using Gemini (#790)



---------

Co-authored-by: Guillaume Vernade <Giom-V@users.noreply.github.com>

* Add service account auth section with IAM roles for Gemini, File, and… (#1258)

* Add service account auth section with IAM roles for Gemini, File, and Vertex AI APIs

* Remove service account section, add link to GCP auth docs instead

* Add Vertex AI authentication documentation link

* Fix embedding types & casing  (#1032)

* Fix embedding types & casing (fixes #1030)

* Fix query task_type and indentation style violations

* fix indentation style

* fixed the formatting and linting issue

* update model ID

* fix formatting

* Update styleguide with API key and crypto restrictions

Added guidelines for API key usage and restrictions on crypto topics.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kkorpal <kkorpal@google.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Shipra <ships3040@gmail.com>
Co-authored-by: Shipra <Shi-pra-19@users.noreply.github.com>
Co-authored-by: Rohan Dwivedi <165134541+ROHANDWIVEDI2005@users.noreply.github.com>
Co-authored-by: AgentModuleAdmin <0agentmodule@gmail.com>
Co-authored-by: Patrick Loeber <50772274+patrickloeber@users.noreply.github.com>
Co-authored-by: Irfaan Mansoori <irfaanmansoori100@gmail.com>
Co-authored-by: Aryan Motgi <85900811+aryanmotgi@users.noreply.github.com>
Co-authored-by: Thor 雷神 Schaeff <5748289+thorwebdev@users.noreply.github.com>
Co-authored-by: Mark Daoust <markdaoust@google.com>
Co-authored-by: August Cayzer <august@cayzer.me>
Co-authored-by: Sri Ujwal <113922053+sbeeredd04@users.noreply.github.com>
Co-authored-by: Sri Ujwal <sbeeredd04@users.noreply.github.com>
Co-authored-by: qianzhu / Lucien <2729478828@qq.com>
Co-authored-by: Naveen Kumar G <naveen13524g@gmail.com>
Co-authored-by: Nitin Tiwari <37960279+NSTiwari@users.noreply.github.com>
Co-authored-by: Anika Islam <172253577+Surjasa@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:quickstarts Issues/PR referencing quickstarts folder status:awaiting review PR awaiting review from a maintainer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Live API does not perceive visual information and hallucinates

4 participants