diff --git a/.github/scripts/spellcheck_conf/wordlist.txt b/.github/scripts/spellcheck_conf/wordlist.txt index cc882099f..52e2e0c83 100644 --- a/.github/scripts/spellcheck_conf/wordlist.txt +++ b/.github/scripts/spellcheck_conf/wordlist.txt @@ -1546,3 +1546,4 @@ AGI DeepEval SDV sklearn +GCP diff --git a/3p-integrations/gcp/README.md b/3p-integrations/gcp/README.md new file mode 100644 index 000000000..dd347f60f --- /dev/null +++ b/3p-integrations/gcp/README.md @@ -0,0 +1,4 @@ +In this folder, we show various recipes for Llama models working with GCP. This currently includes: +* Examples for running Llama 4 model inference on Vertex's serverless API offerings (aka. MaaS) + * tool calling + * JSON mode (structured outputs) diff --git a/3p-integrations/gcp/vertex_MaaS/Vertex_JSON_mode_for_Llama_4.ipynb b/3p-integrations/gcp/vertex_MaaS/Vertex_JSON_mode_for_Llama_4.ipynb new file mode 100644 index 000000000..b94219fa8 --- /dev/null +++ b/3p-integrations/gcp/vertex_MaaS/Vertex_JSON_mode_for_Llama_4.ipynb @@ -0,0 +1,275 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "lsCK4KDHw4LK" + }, + "source": [ + "# **Why JSON mode?**\n", + "\n", + "You can guarantee that a model's generated output always adheres to a specific schema so that you receive consistently formatted responses. For example, you might have an established data schema that you use for other tasks. If you have the model follow the same schema, you can directly extract data from the model's output without any post-processing.\n", + "\n", + "To specify the structure of a model's output, define a response schema, which works like a blueprint for model responses. When you submit a prompt and include the response schema, the model's response always follows your defined schema.\n", + "\n", + "# **Objectives**\n", + "\n", + "In this tutorial, you will learn how to use either OpenAI SDK or Vertex AI SDK in Python to generated structured outputs via the Llama 4 Maverick fully managed model on Vertex AI.\n", + " See here for more info on using the [OpenAI SDK with Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/openai/overview#:~:text=The%20Chat%20Completions%20API%20works,the%20Google%20Gen%20AI%20SDK.), as well as recommendations on when to use OpenAI SDK vs. Vertex AI SDK.\n", + "\n", + "We will use sentiment analysis as an example use case, you can replace it with a different structure that's right for you.\n", + "\n", + "# **Setup and Relevant Links**\n", + "Llama on Vertex AI (fully managed): https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/llama. You may also need to accept the EULA to continue.\n", + "\n", + "Official docs from Vertex on structured outputs/JSON mode with Llama coming soon.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aEOMDh39uMYV" + }, + "source": [ + "# **Setup and Relevant Links **" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "2sHCLvoCw6TA" + }, + "outputs": [], + "source": [ + "import sys\n", + "import IPython\n", + "\n", + "%pip install --upgrade --user --quiet google-genai\n", + "app = IPython.Application.instance()\n", + "app.kernel.do_shutdown(True)\n", + "\n", + "if \"google.colab\" in sys.modules:\n", + " from google.colab import auth\n", + "\n", + " auth.authenticate_user()" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "id": "JG5BoLBdw5je" + }, + "outputs": [], + "source": [ + "import os\n", + "\n", + "project_id = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n", + "if not project_id or project_id == \"[your-project-id]\":\n", + " project_id = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", + "\n", + "# run gcloud auth print-access-token from terminal to get this\n", + "access_token = \"\"\n", + "\n", + "location = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-east5\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DsRB381L0bwb" + }, + "source": [ + "Defining the format we want to return..." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "7CyBOejOz8iR" + }, + "outputs": [], + "source": [ + "response_schema = {\n", + " \"type\": \"ARRAY\",\n", + " \"items\": {\n", + " \"type\": \"ARRAY\",\n", + " \"items\": {\n", + " \"type\": \"OBJECT\",\n", + " \"properties\": {\n", + " \"rating\": {\"type\": \"INTEGER\"},\n", + " \"flavor\": {\"type\": \"STRING\"},\n", + " \"sentiment\": {\n", + " \"type\": \"STRING\",\n", + " \"enum\": [\"POSITIVE\", \"NEGATIVE\", \"NEUTRAL\"],\n", + " },\n", + " \"explanation\": {\"type\": \"STRING\"},\n", + " },\n", + " \"required\": [\"rating\", \"flavor\", \"sentiment\", \"explanation\"],\n", + " },\n", + " },\n", + "}\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pRym0HCpt99W" + }, + "source": [ + "# **Now, generate JSON output**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wSTscqE-wxyd" + }, + "source": [ + "First, with OpenAI's SDK..." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "yU25L2KDpMSG", + "outputId": "c7f8f0c7-3262-434b-f713-8f002d97dd0f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'reviews': [{'text': \"Absolutely loved it! Best ice cream I've ever had.\", 'rating': 4, 'flavor': 'Strawberry Cheesecake', 'sentiment': 'Positive', 'explanation': \"The reviewer uses the phrase 'Absolutely loved it' and states it's the 'Best ice cream I've ever had', indicating a very positive sentiment. The rating of 4 out of a presumed 5 is consistent with this positive sentiment.\"}, {'text': 'Quite good, but a bit too sweet for my taste.', 'rating': 1, 'flavor': 'Mango Tango', 'sentiment': 'Negative', 'explanation': \"Although the reviewer starts with 'Quite good', they follow it with a negative statement 'but a bit too sweet for my taste', indicating a mixed sentiment. However, the rating of 1 suggests a strongly negative sentiment, which is inconsistent with the text. The sentiment classification based on the text would be 'Mixed' or 'Neutral', but given the low rating, it leans more towards being negative overall.\"}]}\n" + ] + } + ], + "source": [ + "import json\n", + "import openai\n", + "from openai import OpenAI\n", + "\n", + "# Setup client\n", + "base_url = f\"https://{location}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{location}/endpoints/openapi\"\n", + "client = openai.OpenAI(base_url=base_url, api_key=access_token)\n", + "\n", + "prompt = \"\"\"\n", + " Analyze the following product reviews, output the sentiment classification, and give an explanation.\n", + "\n", + " - \"Absolutely loved it! Best ice cream I've ever had.\" Rating: 4, Flavor: Strawberry Cheesecake\n", + " - \"Quite good, but a bit too sweet for my taste.\" Rating: 1, Flavor: Mango Tango\n", + "\"\"\"\n", + "\n", + "response = client.chat.completions.create(\n", + " model=\"meta/llama-4-maverick-17b-128e-instruct-maas\",\n", + " messages=[{\"role\": \"user\", \"content\": prompt}],\n", + " response_format={\"type\": \"json_object\"},\n", + " temperature=0.1\n", + ")\n", + "\n", + "product_reviews = json.loads(response.choices[0].message.content)\n", + "print(product_reviews)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ckdF273K0g7E" + }, + "source": [ + "Now with Vertex AI SDK" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "OnfgZyHZr9gl" + }, + "outputs": [], + "source": [ + "from google.genai.types import GenerateContentConfig, Part, SafetySetting\n", + "\n", + "from google import genai\n", + "\n", + "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)\n", + "MODEL_ID = \"meta/llama-4-maverick-17b-128e-instruct-maas\"\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Z_ka6SP90Am-", + "outputId": "69d91ee1-96d4-44f9-d22d-c4f4a9887e82" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[{'explanation': \"The reviewer used the phrase 'Absolutely loved it' and stated it was the 'Best ice cream I've ever had', indicating a very positive sentiment despite the rating being less than 5.\", 'flavor': 'Strawberry Cheesecake', 'rating': 4, 'sentiment': 'POSITIVE'}, {'explanation': \"The reviewer described the product as 'Quite good', but expressed a negative aspect by stating it was 'a bit too sweet', aligning with the low rating given. The negative aspect outweighs the positive, leading to an overall negative sentiment.\", 'flavor': 'Mango Tango', 'rating': 1, 'sentiment': 'NEGATIVE'}]]\n" + ] + } + ], + "source": [ + "prompt = \"\"\"\n", + " Analyze the following product reviews, output the sentiment classification, and give an explanation.\n", + "\n", + " - \"Absolutely loved it! Best ice cream I've ever had.\" Rating: 4, Flavor: Strawberry Cheesecake\n", + " - \"Quite good, but a bit too sweet for my taste.\" Rating: 1, Flavor: Mango Tango\n", + "\"\"\"\n", + "\n", + "response = client.models.generate_content(\n", + " model=MODEL_ID,\n", + " contents=prompt,\n", + " config=GenerateContentConfig(\n", + " response_mime_type=\"application/json\",\n", + " response_schema=response_schema,\n", + " ),\n", + ")\n", + "product_reviews: dict = response.parsed\n", + "print(product_reviews)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wC4ci2UEF9KB" + }, + "source": [ + "# **Congrats and conclusion**\n", + "\n", + "You've successfully built a sentiment analyzer leveraging structured outputs via Llama 4 Maverick using the OpenAI and/or Vertex AI SDK!\n", + "\n", + "# **Cleanup**\n", + "You can perform the following cleanup to avoid incurring charges to your Google Cloud account for the resources used in this codelab:\n", + "* To avoid unnecessary Google Cloud charges, use the Google Cloud console to delete your project if you do not need it.\n", + "* If you want to disable the APIs for Vertex AI, navigate to the Vertex AI API Service Details page and click Disable API and confirm.\n" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/3p-integrations/gcp/vertex_MaaS/Vertex_tool_calling_for_Llama_4.ipynb b/3p-integrations/gcp/vertex_MaaS/Vertex_tool_calling_for_Llama_4.ipynb new file mode 100644 index 000000000..c1d4eed86 --- /dev/null +++ b/3p-integrations/gcp/vertex_MaaS/Vertex_tool_calling_for_Llama_4.ipynb @@ -0,0 +1,497 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# **Why function calling?**\n", + "\n", + "Imagine asking someone to write down important information without giving them a form or any guidelines on the structure. You might get a beautifully crafted paragraph, but extracting specific details like names, dates, or numbers would be tedious! Similarly, trying to get consistent structured data from a generative text model without function calling can be frustrating. You're stuck explicitly prompting for things like JSON output, often with inconsistent and frustrating results.\n", + "\n", + "This is where function calling comes in. Instead of hoping for the best in a freeform text response from a generative model, you can define clear functions with specific parameters and data types. These function declarations act as structured guidelines, guiding the Llama model to structure its output in a predictable and usable way. No more parsing text responses for important information!\n", + "\n", + "Think of it like teaching Llama to speak the language of your applications. Need to retrieve information from a database? Define a search_db function with parameters for search terms. Want to integrate with a weather API? Create a get_weather function that takes a location as input. Function calling bridges the gap between human language and the structured data needed to interact with external systems.\n", + "\n", + "# **Objectives**\n", + "\n", + "In this tutorial, you will learn how to use either OpenAI SDK or Vertex AI SDK in Python to make function calls via the Llama 4 Maverick fully managed model on Vertex AI.\n", + " See here for more info on using the [OpenAI SDK with Vertex](https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/openai/overview#:~:text=The%20Chat%20Completions%20API%20works,the%20Google%20Gen%20AI%20SDK.), as well as recommendations on when to use OpenAI SDK vs. Vertex AI SDK.\n", + "\n", + "We will use a currency exchange function as an example, you can replace it with another function with the right functionality for you.\n", + "This tutorial is based on this Vertex AI codelab: https://codelabs.developers.google.com/codelabs/gemini-function-calling\n", + "\n", + "# **Setup and Relevant Links**\n", + "Llama on Vertex AI (fully managed): https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/llama\n", + "\n", + "Official docs from Vertex on tool calling with Llama coming soon.\n" + ], + "metadata": { + "id": "JIVs8gXB1Auw" + } + }, + { + "cell_type": "markdown", + "source": [ + "# **First, with OpenAI's SDK**" + ], + "metadata": { + "id": "1zikcmKb5JVx" + } + }, + { + "cell_type": "markdown", + "source": [ + "Handling imports and setup" + ], + "metadata": { + "id": "8znWkSqT5zyy" + } + }, + { + "cell_type": "code", + "source": [ + "import google.auth\n", + "import openai\n", + "import json\n", + "\n", + "from google.auth.transport import requests as google_requests" + ], + "metadata": { + "id": "C81B_THj5XUZ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "import os\n", + "\n", + "project_id = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n", + "if not project_id or project_id == \"[your-project-id]\":\n", + " project_id = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", + "\n", + "location = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-east5\")\n", + "\n", + "# run gcloud auth print-access-token from terminal to get this\n", + "access_token = \"\"\n", + "\n", + "# Set up OpenAI client\n", + "base_url = f\"https://{location}-aiplatform.googleapis.com/v1/projects/{project_id}/locations/{location}/endpoints/openapi\"\n", + "client = openai.OpenAI(base_url=base_url, api_key=access_token)" + ], + "metadata": { + "id": "ivcFdple7Qr5" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "define the function implementation" + ], + "metadata": { + "id": "wqhyLJO07Gaz" + } + }, + { + "cell_type": "code", + "source": [ + "def get_exchange_rate(currency_date, currency_from, currency_to):\n", + " \"\"\"Get the exchange rate for currencies between countries\"\"\"\n", + " try:\n", + " url = f\"https://api.frankfurter.app/{currency_date}\"\n", + " params = {\n", + " \"from\": currency_from,\n", + " \"to\": currency_to\n", + " }\n", + " response = requests.get(url, params=params)\n", + " response.raise_for_status()\n", + " return json.dumps(response.json())\n", + " except Exception as e:\n", + " return f\"Error fetching exchange rate: {str(e)}\"" + ], + "metadata": { + "id": "4JagRT1t7DT8" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "set up the function declaration for the model" + ], + "metadata": { + "id": "LmS_8mdenPEU" + } + }, + { + "cell_type": "code", + "source": [ + "tools = [{\n", + " \"type\": \"function\",\n", + " \"function\": {\n", + " \"name\": \"get_exchange_rate\",\n", + " \"description\": \"Get the exchange rate for currencies between countries\",\n", + " \"parameters\": {\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"currency_date\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"A date in YYYY-MM-DD format or 'latest'\"\n", + " },\n", + " \"currency_from\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The currency to convert from in ISO 4217 format\"\n", + " },\n", + " \"currency_to\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The currency to convert to in ISO 4217 format\"\n", + " }\n", + " },\n", + " \"required\": [\"currency_from\", \"currency_to\"]\n", + " }\n", + " }\n", + "}]" + ], + "metadata": { + "id": "4lP5F51Cmq7f" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "set up the client, generate a tool call and execute the tool call\n", + "Enter a query into the text field to get started, eg. \"100 usd to eur\"" + ], + "metadata": { + "id": "iCGAtoNwmlFc" + } + }, + { + "cell_type": "code", + "source": [ + "def main():\n", + " # Set up client\n", + " client = openai.OpenAI(base_url=base_url, api_key=access_token)\n", + "\n", + " # Step 1: Send the user query to the model\n", + " user_query = input(\"Enter your currency exchange query: \")\n", + "\n", + " completion = client.chat.completions.create(\n", + " model=\"meta/llama-4-maverick-17b-128e-instruct-maas\",\n", + " messages=messages,\n", + " tools=tools,\n", + " temperature=0.1,\n", + " max_tokens=512,\n", + " tool_choice=\"auto\"\n", + " )\n", + "\n", + " # Step 2: Check if the model called a function\n", + " if completion.choices[0].message.tool_calls:\n", + " # Step 3: Execute the function\n", + " for tool_call in completion.choices[0].message.tool_calls:\n", + " if tool_call.function.name == \"get_exchange_rate\":\n", + " args = json.loads(tool_call.function.arguments)\n", + " currency_date = args.get(\"currency_date\", \"latest\")\n", + " currency_from = args.get(\"currency_from\")\n", + " currency_to = args.get(\"currency_to\")\n", + " result = get_exchange_rate(currency_date, currency_from, currency_to)\n", + "\n", + " # Add model's response with function call\n", + " messages.append(completion.choices[0].message)\n", + " messages.append({\n", + " \"role\": \"tool\",\n", + " \"tool_call_id\": tool_call.id,\n", + " \"content\": result\n", + " })\n", + "\n", + " # Get final response from model\n", + " final_response = client.chat.completions.create(\n", + " model=\"meta/llama-4-maverick-17b-128e-instruct-maas\",\n", + " messages=messages,\n", + " temperature=0.1\n", + " )\n", + "\n", + " print(\"Final response:\", final_response.choices[0].message.content)\n", + " else:\n", + " print(\"Direct response:\", completion.choices[0].message.content)\n", + "\n", + "if __name__ == \"__main__\":\n", + " main()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "SycilP3AkiRr", + "outputId": "f3bcab82-9010-4610-d392-f36792c57844" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Enter your currency exchange query: 100 usd to eur\n", + "Direct response: The exchange rate for 100 USD to EUR is 88.645 EUR.\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "# **Now with Vertex SDK**" + ], + "metadata": { + "id": "kyp2Fy_7noaV" + } + }, + { + "cell_type": "markdown", + "source": [ + "Handling imports and setup" + ], + "metadata": { + "id": "KucVr8Txplnz" + } + }, + { + "cell_type": "code", + "source": [ + "import json\n", + "import requests\n", + "import vertexai\n", + "from vertexai.generative_models import (\n", + " FunctionDeclaration,\n", + " GenerationConfig,\n", + " GenerativeModel,\n", + " Tool,\n", + " Content,\n", + " Part\n", + ")\n", + "\n", + "# Authenticate with Google Cloud\n", + "from google.colab import auth\n", + "auth.authenticate_user()" + ], + "metadata": { + "id": "JxVuZ3R_pmKH" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# Initialize Vertex AI\n", + "import os\n", + "\n", + "PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n", + "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n", + " PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n", + "\n", + "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n", + "\n", + "vertexai.init(project=PROJECT_ID, location=LOCATION)" + ], + "metadata": { + "id": "r1rP7PdUpwK6" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "define the function implementation" + ], + "metadata": { + "id": "nFFEsV1BptAD" + } + }, + { + "cell_type": "code", + "source": [ + "def get_exchange_rate(currency_date, currency_from, currency_to):\n", + " \"\"\"Get the exchange rate for currencies between countries\"\"\"\n", + " try:\n", + " url = f\"https://api.frankfurter.app/{currency_date}\"\n", + " params = {\n", + " \"from\": currency_from,\n", + " \"to\": currency_to\n", + " }\n", + " response = requests.get(url, params=params)\n", + " response.raise_for_status()\n", + " return response.json()\n", + " except Exception as e:\n", + " return {\"error\": str(e)}" + ], + "metadata": { + "id": "hDMzxUBxptq0" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "set up the function declaration for the model" + ], + "metadata": { + "id": "g1RH1tplqhi5" + } + }, + { + "cell_type": "code", + "source": [ + "get_exchange_rate_func = FunctionDeclaration(\n", + " name=\"get_exchange_rate\",\n", + " description=\"Get the exchange rate for currencies between countries\",\n", + " parameters={\n", + " \"type\": \"object\",\n", + " \"properties\": {\n", + " \"currency_date\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"A date in YYYY-MM-DD format or 'latest'\"\n", + " },\n", + " \"currency_from\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The currency to convert from in ISO 4217 format\"\n", + " },\n", + " \"currency_to\": {\n", + " \"type\": \"string\",\n", + " \"description\": \"The currency to convert to in ISO 4217 format\"\n", + " }\n", + " },\n", + " \"required\": [\"currency_from\", \"currency_to\"]\n", + " },\n", + ")\n", + "\n", + "exchange_tool = Tool(\n", + " function_declarations=[get_exchange_rate_func],\n", + ")\n" + ], + "metadata": { + "id": "XyR1QzkSozrq" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "set up the client, generate a tool call and execute the tool call\n", + "Enter a query into the text field to get started, eg. \"100 usd to eur\"" + ], + "metadata": { + "id": "MFM6gfqwqzsi" + } + }, + { + "cell_type": "code", + "source": [ + "model = GenerativeModel(\n", + " \"llama-4-maverick-17b-128e-instruct-maas\",\n", + " generation_config=GenerationConfig(temperature=0.1),\n", + " tools=[exchange_tool],\n", + ")\n", + "\n", + "def main():\n", + " user_query = input(\"Enter your currency exchange query: \")\n", + "\n", + " # Start chat and send user query\n", + " chat = model.start_chat()\n", + " response = chat.send_message(user_query)\n", + "\n", + " # Check for function calls\n", + " if response.candidates[0].function_calls:\n", + " for function_call in response.candidates[0].function_calls:\n", + " if function_call.name == \"get_exchange_rate\":\n", + " # Extract arguments\n", + " args = function_call.args\n", + " currency_date = args.get(\"currency_date\", \"latest\")\n", + " currency_from = args.get(\"currency_from\")\n", + " currency_to = args.get(\"currency_to\")\n", + "\n", + " # Call the function\n", + " result = get_exchange_rate(currency_date, currency_from, currency_to)\n", + "\n", + " # Send function result back to model\n", + " final_response = chat.send_message(\n", + " Content(\n", + " role=\"function\",\n", + " parts=[\n", + " Part.from_function_response(\n", + " name=\"get_exchange_rate\",\n", + " response=result\n", + " )\n", + " ]\n", + " )\n", + " )\n", + "\n", + " print(\"Final response:\", final_response.text)\n", + " else:\n", + " print(\"Direct response:\", response.text)\n", + "\n", + "if __name__ == \"__main__\":\n", + " main()\n" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "L_Dla5r6qyrV", + "outputId": "bc2e9eae-11cc-4095-c822-85edcf7b703e" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Enter your currency exchange query: 100 usd to eur\n", + "Final response: The exchange rate for 100 USD to EUR on the latest date is 88.645 EUR.\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "# **Congrats and conclusion**\n", + "\n", + "Leveraging function calling via Llama 4 in Vertex AI, you've successfully built a generative AI pipeline that uses the OpenAI and/or Vertex AI SDK! Users can ask about exchange rates, and the system will fetch the latest data from an external API and respond with an answer.\n", + "\n", + "Given a prompt from an end-user, Llama takes care of selecting the appropriate function, extracting parameters from the prompt, and returning a structured data object for you to make an external API call.\n", + "\n", + "\n", + "# **Cleanup**\n", + "You can perform the following cleanup to avoid incurring charges to your Google Cloud account for the resources used in this codelab:\n", + "* To avoid unnecessary Google Cloud charges, use the Google Cloud console to delete your project if you do not need it.\n", + "* If you want to disable the APIs for Vertex AI, navigate to the Vertex AI API Service Details page and click Disable API and confirm.\n" + ], + "metadata": { + "id": "pT9Cf6TtAgX7" + } + } + ] +} \ No newline at end of file