|
| 1 | +{ |
| 2 | + "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "code", |
| 5 | + "execution_count": null, |
| 6 | + "metadata": { |
| 7 | + "cellView": "form", |
| 8 | + "id": "YsFQaflOxP_g" |
| 9 | + }, |
| 10 | + "outputs": [], |
| 11 | + "source": [ |
| 12 | + "# Copyright 2025 Google LLC\n", |
| 13 | + "#\n", |
| 14 | + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", |
| 15 | + "# you may not use this file except in compliance with the License.\n", |
| 16 | + "# You may obtain a copy of the License at\n", |
| 17 | + "#\n", |
| 18 | + "# https://www.apache.org/licenses/LICENSE-2.0\n", |
| 19 | + "#\n", |
| 20 | + "# Unless required by applicable law or agreed to in writing, software\n", |
| 21 | + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", |
| 22 | + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", |
| 23 | + "# See the License for the specific language governing permissions and\n", |
| 24 | + "# limitations under the License." |
| 25 | + ] |
| 26 | + }, |
| 27 | + { |
| 28 | + "cell_type": "markdown", |
| 29 | + "metadata": { |
| 30 | + "id": "Y-uUs1OfxcjA" |
| 31 | + }, |
| 32 | + "source": [ |
| 33 | + "# Vertex AI Model Garden - T5Gemma 2(Local Inference)\n", |
| 34 | + "\n", |
| 35 | + "<table><tbody><tr>\n", |
| 36 | + " <td style=\"text-align: center\">\n", |
| 37 | + " <a href=\"https://console.cloud.google.com/vertex-ai/notebooks/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/model_garden/model_garden_t5gemma_2_local_inference.ipynb\">\n", |
| 38 | + " <img alt=\"Workbench logo\" src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" width=\"32px\"><br> Run in Workbench\n", |
| 39 | + " </a>\n", |
| 40 | + " </td>\n", |
| 41 | + " <td style=\"text-align: center\">\n", |
| 42 | + " <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Fmodel_garden%2Fmodel_garden_t5gemma_2_local_inference.ipynb\">\n", |
| 43 | + " <img alt=\"Google Cloud Colab Enterprise logo\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" width=\"32px\"><br> Run in Colab Enterprise\n", |
| 44 | + " </a>\n", |
| 45 | + " </td>\n", |
| 46 | + " <td style=\"text-align: center\">\n", |
| 47 | + " <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/model_garden/model_garden_t5gemma_2_local_inference.ipynb\">\n", |
| 48 | + " <img alt=\"GitHub logo\" src=\"https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png\" width=\"32px\"><br> View on GitHub\n", |
| 49 | + " </a>\n", |
| 50 | + " </td>\n", |
| 51 | + "</tr></tbody></table>" |
| 52 | + ] |
| 53 | + }, |
| 54 | + { |
| 55 | + "cell_type": "markdown", |
| 56 | + "metadata": { |
| 57 | + "id": "-c_LNERL0MNv" |
| 58 | + }, |
| 59 | + "source": [ |
| 60 | + "## Overview\n", |
| 61 | + "\n", |
| 62 | + "This notebook demonstrates how to install the necessary libraries and run local inference with T5Gemma 2 model in a [Colab Enterprise Instance](https://cloud.google.com/colab/docs) or a [Workbench Instance](https://cloud.google.com/vertex-ai/docs/workbench/instances).\n", |
| 63 | + "\n", |
| 64 | + "### Objective\n", |
| 65 | + "\n", |
| 66 | + "Run local inference with T5Gemma 2 model.\n", |
| 67 | + "\n", |
| 68 | + "### Costs\n", |
| 69 | + "\n", |
| 70 | + "This tutorial uses billable components of Google Cloud:\n", |
| 71 | + "\n", |
| 72 | + "* Vertex AI\n", |
| 73 | + "\n", |
| 74 | + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage." |
| 75 | + ] |
| 76 | + }, |
| 77 | + { |
| 78 | + "cell_type": "markdown", |
| 79 | + "metadata": { |
| 80 | + "id": "zsNTHGCK1FU7" |
| 81 | + }, |
| 82 | + "source": [ |
| 83 | + "## Install dependencies\n", |
| 84 | + "\n", |
| 85 | + "Before you begin, make sure you are using an instance with GPU.\n", |
| 86 | + "\n", |
| 87 | + "* **Colab Enterprise**: Connect to a [Colab Enterprise runtime](https://cloud.google.com/colab/docs/connect-to-runtime) with GPU. If not, we recommend [creating a runtime template](https://cloud.google.com/colab/docs/create-runtime-template) with `g2-standard-16` machine type (or larger, see the descriptions of the model you want to try out below) to use `NVIDIA_L4` GPU. Then, [create a runtime](https://cloud.google.com/colab/docs/create-runtime) from that template.\n", |
| 88 | + "\n", |
| 89 | + "* **Workbench Instance**: Use a Workbench Instance with GPU." |
| 90 | + ] |
| 91 | + }, |
| 92 | + { |
| 93 | + "cell_type": "code", |
| 94 | + "execution_count": null, |
| 95 | + "metadata": { |
| 96 | + "cellView": "form", |
| 97 | + "id": "xle1_0ns1r10" |
| 98 | + }, |
| 99 | + "outputs": [], |
| 100 | + "source": [ |
| 101 | + "!pip install -q git+https://github.com/huggingface/transformers.git\n", |
| 102 | + "!pip install torch~=2.8.0\n", |
| 103 | + "!pip install torchvision~=0.23.0" |
| 104 | + ] |
| 105 | + }, |
| 106 | + { |
| 107 | + "cell_type": "markdown", |
| 108 | + "metadata": { |
| 109 | + "id": "a0eE5dWx1s1j" |
| 110 | + }, |
| 111 | + "source": [ |
| 112 | + "## Hugging Face Login\n", |
| 113 | + "\n", |
| 114 | + "The following code block will prompt you to enter your Hugging Face access token.\n", |
| 115 | + "\n", |
| 116 | + "If you don't already have a Hugging Face access token, follow the [Hugging Face documentation](https://huggingface.co/docs/hub/en/security-tokens) to create an access token with \"read\" permission. You can find your existing access tokens in the Hugging Face [Access Token](https://huggingface.co/settings/tokens) page.\n", |
| 117 | + "\n", |
| 118 | + "Make sure you have accepted the model agreement to access the model." |
| 119 | + ] |
| 120 | + }, |
| 121 | + { |
| 122 | + "cell_type": "code", |
| 123 | + "execution_count": null, |
| 124 | + "metadata": { |
| 125 | + "cellView": "form", |
| 126 | + "id": "oR2eJczK3mNF" |
| 127 | + }, |
| 128 | + "outputs": [], |
| 129 | + "source": [ |
| 130 | + "from huggingface_hub import notebook_login\n", |
| 131 | + "\n", |
| 132 | + "notebook_login()" |
| 133 | + ] |
| 134 | + }, |
| 135 | + { |
| 136 | + "cell_type": "markdown", |
| 137 | + "metadata": { |
| 138 | + "id": "arlTS97v29Uu" |
| 139 | + }, |
| 140 | + "source": [ |
| 141 | + "## Run local inference" |
| 142 | + ] |
| 143 | + }, |
| 144 | + { |
| 145 | + "cell_type": "code", |
| 146 | + "execution_count": null, |
| 147 | + "metadata": { |
| 148 | + "cellView": "form", |
| 149 | + "id": "uTLuTO3N1z_B" |
| 150 | + }, |
| 151 | + "outputs": [], |
| 152 | + "source": [ |
| 153 | + "import requests\n", |
| 154 | + "from PIL import Image\n", |
| 155 | + "from transformers import AutoModelForSeq2SeqLM, AutoProcessor\n", |
| 156 | + "\n", |
| 157 | + "model_id = \"google/t5gemma-2-270m-270m\" # @param [\"google/t5gemma-2-270m-270m\", \"google/t5gemma-2-1b-1b\", \"google/t5gemma-2-4b-4b\"] {isTemplate:true}\n", |
| 158 | + "\n", |
| 159 | + "processor = AutoProcessor.from_pretrained(model_id)\n", |
| 160 | + "model = AutoModelForSeq2SeqLM.from_pretrained(model_id)\n", |
| 161 | + "\n", |
| 162 | + "url = \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"\n", |
| 163 | + "image = Image.open(requests.get(url, stream=True).raw)\n", |
| 164 | + "prompt = \"<start_of_image> in this image, there is\"\n", |
| 165 | + "\n", |
| 166 | + "model_inputs = processor(text=prompt, images=image, return_tensors=\"pt\")\n", |
| 167 | + "generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)\n", |
| 168 | + "print(processor.decode(generation[0]))" |
| 169 | + ] |
| 170 | + } |
| 171 | + ], |
| 172 | + "metadata": { |
| 173 | + "colab": { |
| 174 | + "name": "model_garden_t5gemma_2_local_inference.ipynb", |
| 175 | + "toc_visible": true |
| 176 | + }, |
| 177 | + "kernelspec": { |
| 178 | + "display_name": "Python 3", |
| 179 | + "name": "python3" |
| 180 | + } |
| 181 | + }, |
| 182 | + "nbformat": 4, |
| 183 | + "nbformat_minor": 0 |
| 184 | +} |
0 commit comments