bigquery_agent_analytics_plugin - Agent Engine (Not Working)

[dvdeveloper]

Hello, I am trying to implement bigquery_agent_analytics_plugin using agent engine. The SA has the permissions, but I cannot get the logs to be recorded with this new implementation. Has anyone been able to resolve this? Cloud Run works, but not agent engine. I am leaving some sample code in case anyone can help me.

agent.py

from google.adk.agents.llm_agent import Agent
from google.adk.models.google_llm import Gemini
from log.registro  import before_agent_callback, after_agent_callback, before_model_callback, after_model_callback , after_tool_callback
from log.langfuse_config import prompt
from google.genai import types
from google.adk.tools.retrieval import VertexAiRagRetrieval
from vertexai import rag
from utils.template_utils import escape_template_variables
from google.adk.tools.bigquery import BigQueryToolset, BigQueryCredentialsConfig


# --- BigQuery Agent Analytics (logs) ---

import os
import logging
from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin,
    BigQueryLoggerConfig,
)

BQ_PROJECT_ID = "tXXXXXX"
BQ_DATASET_ID = "agent_engine"
BQ_TABLE_ID = "XXXXX"


bq_logging_plugin = BigQueryAgentAnalyticsPlugin(
    project_id=BQ_PROJECT_ID,
    dataset_id=BQ_DATASET_ID,
    table_id=BQ_TABLE_ID,
    location="us-central1",  # Especificar location explícitamente
)

"""
credentials, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
bigquery_toolset = BigQueryToolset(
    credentials_config=BigQueryCredentialsConfig(credentials=credentials)
)
 """

PROMPT = escape_template_variables(prompt.prompt)
LANGFUSE_AGENT_NAME = prompt.config['AGENT_NAME']
MODEL_NAME = prompt.config['MODEL']

PROJECT_ID = prompt.config['PROJECT_ID']
LOCATION = prompt.config['LOCATION']
CORPUS_ID = prompt.config['CORPUS_ID']
RAG_CORPUS_ID = f"projects/{PROJECT_ID}/locations/{LOCATION}/ragCorpora/{CORPUS_ID}"

rag_tool = VertexAiRagRetrieval(
    name="BaseConocimiento",
    description="Base de datos de conocimiento asociado Dexter Agente",
    rag_resources=[rag.RagResource(rag_corpus=RAG_CORPUS_ID)],
)

root_agent = Agent(
    name=LANGFUSE_AGENT_NAME,
    model=MODEL_NAME,
    instruction=PROMPT,
    tools=[rag_tool],
    before_agent_callback=before_agent_callback,
    after_agent_callback=after_agent_callback,
    before_model_callback=before_model_callback,
    after_model_callback=after_model_callback,
    after_tool_callback=after_tool_callback,
)

update.py

import vertexai
from vertexai import agent_engines
from vertexai.preview import reasoning_engines
from agente.agent import root_agent, bq_logging_plugin
import uuid

vertexai.init(
    project="XXXXXXXXX",
    location="us-central1",
    staging_bucket="gs://XXXXXXXXX"
)

# AdkApp acepta agent y plugins por separado
app_adk = reasoning_engines.AdkApp(
    agent=root_agent,
    plugins=[bq_logging_plugin],
    enable_tracing=True,
)

gcs_dir_name = str(uuid.uuid4())
remote_app = agent_engines.update(
    agent_engine=app_adk,
    resource_name="projects/XXXXX/locations/us-central1/reasoningEngines/X06XXX68445XXX70XXXX",
    service_account="sa-agents@XXXXXX",
    requirements=[
        "google-adk>=1.21.0",
        "google-cloud-aiplatform[adk,agent_engines]>=1.21.0",
        "llama-index",
        "langfuse",
        "google-cloud-secret-manager",
        "google-cloud-firestore",
        "google-cloud-bigquery",
        "google-cloud-bigquery-storage",
        "google-cloud-storage",
    ],
    extra_packages=["./agente","./log","./utils"],
    gcs_dir_name=f"tXXXXX{gcs_dir_name}",
    env_vars={
        "GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY": "true",
        "OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT": "true",
    }
)

[haiyuan-eng-google]

Hi @dvdeveloper thanks for reporting this bug, can you do below steps to help figure out the root cause?

Check Cloud Logging: Go to the Google Cloud Console, navigate to Logs Explorer, and filter by resource.type="reasoning_engine". Look for any "Permission Denied" or "Field not found" errors related to BigQuery or the ADK internal processes.

Version Match: Ensure that the google-adk and google-cloud-aiplatform versions are consistent between your local development (where Cloud Run likely works) and the requirements you've listed in update.py. Also make sure google-adk version running on Agent Engine is >= 1.21.0

[Dustan-FullStory]

@haiyuan-eng-google : That's fantastic to hear, thank you so much for digging into this and shipping a fix so quickly!

[haiyuan-eng-google]

@Dustan-FullStory can you try below steps to make sure the new version really fix your issue? Thanks

1. upgrade their Python ADK to the latest version 1.23.0,
2. redeploy their agent (make sure the deployed environment used ADK 1.23.0)
3. then test to confirm the fix is working ?

[Dustan-FullStory]

I have re-deployed the agent using ADK 1.23.0 and unfortunately I am still not seeing analytics data being sent to BQ from Agent Engine after a test interaction with the agent.

{'http_options': {'headers': {'x-goog-api-client': 'google-adk/1.23.0+remote_reasoning_engine gl-python/3.13.11', 'user-agent': 'google-adk/1.23.0+remote_reasoning_engine gl-python/3.13.11'}}, 'tools': [{}], 'labels': {'adk_agent_name': 'fs_AIShaman_agent'}}

The behavior seems to be the same as before, I have the debug logging enabled still and I can see all of the agent interactions in the Cloud Logging console, but there are no errors or anything related to permissions that I can see. It's just like it does not try to use the plugin though I am certain it is being loaded, I added some additional logging of the plugin list when the adkapp is created:

INFO 2026-01-23T17:36:07.800535Z App created successfully. App.plugins: [<google.adk.plugins.bigquery_agent_analytics_plugin.BigQueryAgentAnalyticsPlugin object at 0x7fb2cd606a50>]

I ran this same agent using 1.23.0 locally in adk web and again everything is working perfectly. Events show up immediately in BigQuery.

I have wrapped all of the plugin callbacks with additional logging too, using this code:

# Wrap plugin callbacks to add logging
if bq_analytics_plugin:
    original_before_run = bq_analytics_plugin.before_run_callback
    original_after_run = bq_analytics_plugin.after_run_callback
    original_before_agent = bq_analytics_plugin.before_agent_callback
    original_after_agent = bq_analytics_plugin.after_agent_callback

    async def logged_before_run(*args, **kwargs):
        logger.info("🔵 BigQuery Plugin: before_run_callback FIRED")
        try:
            result = await original_before_run(*args, **kwargs)
            logger.info("🔵 BigQuery Plugin: before_run_callback COMPLETED")
            return result
        except Exception as e:
            logger.error(f"🔴 BigQuery Plugin: before_run_callback ERROR: {e}", exc_info=True)
            raise

    async def logged_after_run(*args, **kwargs):
        logger.info("🟢 BigQuery Plugin: after_run_callback FIRED")
        try:
            result = await original_after_run(*args, **kwargs)
            logger.info("🟢 BigQuery Plugin: after_run_callback COMPLETED")
            return result
        except Exception as e:
            logger.error(f"🔴 BigQuery Plugin: after_run_callback ERROR: {e}", exc_info=True)
            raise

    async def logged_before_agent(*args, **kwargs):
        logger.info("🔵 BigQuery Plugin: before_agent_callback FIRED")
        return await original_before_agent(*args, **kwargs)

    async def logged_after_agent(*args, **kwargs):
        logger.info("🟢 BigQuery Plugin: after_agent_callback FIRED")
        return await original_after_agent(*args, **kwargs)

    bq_analytics_plugin.before_run_callback = logged_before_run
    bq_analytics_plugin.after_run_callback = logged_after_run
    bq_analytics_plugin.before_agent_callback = logged_before_agent
    bq_analytics_plugin.after_agent_callback = logged_after_agent
    logger.info("✅ Wrapped BigQuery plugin callbacks with logging")

When running in local adk web I see the callbacks firing during every invocation as I would expect, but in AgentEngine I don't see them. Only the "Wrapped BigQuery plugin callbacks with logging" log ever appears in Cloud Logging.

What could make the plugin callbacks not fire in AgentEngine, when they work in adk web?

[haiyuan-eng-google]

@Dustan-FullStory Thanks for quick update, can you double check the ADK version you used to deployed in agent engine environment also ADK 1.23.0 ?

[Dustan-FullStory]

Yes, definitely, the log:

{'http_options': {'headers': {'x-goog-api-client': 'google-adk/1.23.0+remote_reasoning_engine gl-python/3.13.11', 'user-agent': 'google-adk/1.23.0+remote_reasoning_engine gl-python/3.13.11'}}, 'tools': [{}], 'labels': {'adk_agent_name': 'fs_AIShaman_agent'}}

Is from the cloud logging console for this agent.

[haiyuan-eng-google]

import os
import shutil
from google.adk.agents import Agent
from google.adk.models.google_llm import Gemini
from google.adk.plugins.bigquery_agent_analytics_plugin import BigQueryAgentAnalyticsPlugin
from google.adk.tools.bigquery import BigQueryCredentialsConfig, BigQueryToolset
import google.auth
import vertexai
from vertexai import agent_engines
# --- Configuration ---
PROJECT_ID = "haiyuan-dev"
LOCATION = "us-central1"
STAGING_BUCKET = "gs://haiyuan-adk-agent-staging-v3"
# ... [Checks for env vars] ...
os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = "True"
BQ_DATASET_ID = "agent_logs"
BQ_TABLE_ID = "0123_invocation_logs_complex"

vertexai.init(
    project=PROJECT_ID, location=LOCATION, staging_bucket=STAGING_BUCKET
)
client = vertexai.Client(project=PROJECT_ID, location=LOCATION)
bq_logger_plugin = BigQueryAgentAnalyticsPlugin(
    project_id=PROJECT_ID,
    dataset_id=BQ_DATASET_ID,
    table_id=BQ_TABLE_ID,
)
try:
  credentials, _ = google.auth.default(
      scopes=["https://www.googleapis.com/auth/cloud-platform"]
  )
except google.auth.exceptions.DefaultCredentialsError:
  raise RuntimeError("Could not get default credentials.")
bq_creds_config = BigQueryCredentialsConfig(credentials=credentials)
bigquery_toolset = BigQueryToolset(credentials_config=bq_creds_config)
llm = Gemini(model="gemini-2.5-flash")


root_agent = Agent(
    model=llm,
    name="my_bq_agent",
    instruction=(
        "You are a helpful assistant. You can use BigQuery tools to answer"
        " questions about data."
    ),
    tools=[bigquery_toolset],
)
app_to_deploy = agent_engines.AdkApp(
    agent=root_agent, plugins=[bq_logger_plugin]
)

# --- 1. Prepare Local Dependencies Directory ---
# Define paths
local_whl_source = "/dist/google_adk-1.23.0-py3-none-any.whl"
dep_dir = "./adk_dependencies"  # Local temporary directory
whl_basename = os.path.basename(local_whl_source)
local_whl_dest = os.path.join(dep_dir, whl_basename)
# Create directory and copy file
if not os.path.exists(local_whl_source):
  raise FileNotFoundError(f"Source wheel not found: {local_whl_source}")
if os.path.exists(dep_dir):
  shutil.rmtree(dep_dir)  # Clean up previous runs
os.makedirs(dep_dir)
shutil.copy(local_whl_source, local_whl_dest)
print(f"Prepared local dependency: {local_whl_dest}")
# --- 2. Deploy ---
print(f"Deploying agent to Vertex AI Agent Engine...")
try:
  remote_app = client.agent_engines.create(
      agent=app_to_deploy,
      config={
          "display_name": "my-adk-app-with-bq-and-logger-v8",
          "staging_bucket": STAGING_BUCKET,
          "requirements": [
              "google-cloud-aiplatform[agent_engines]",
              # Reference the file inside the uploaded directory
              f"adk_dependencies/{whl_basename}",
              "google-cloud-bigquery",
              "google-auth",
              "db-dtypes",
              "pyarrow",
              "pydantic",
              "cloudpickle",
          ],
          # Upload the entire directory. It should appear as /code/adk_dependencies/ remotely.
          "extra_packages": [dep_dir],
      },
  )
  print(f"Deployed agent: {remote_app.api_resource.name}")
except Exception as e:
  print(f"Failed to deploy agent: {e}")
  import traceback

  traceback.print_exc()
finally:
  # Optional: Cleanup the temporary directory
  # if os.path.exists(dep_dir):
  #    shutil.rmtree(dep_dir)
  pass

[Dustan-FullStory]

Okay I've somehow managed to get this working now, but I'm not entirely sure what the real fix was. I think I was using the adk deploy incorrectly, or perhaps in a deprecated manner. There are new flags for adk deploy and I went through and made sure I was using them all.

In particular I think --adk_app_object was the actual fix, but I can't figure out why.

[Dustan-FullStory]

I'm afraid I cannot as it doesn't belong to me. I will say it's really just a simple logic wrapper around the adk deploy command line utililty. It reads the .env file to gather the particular details of the deployment and just execs it using a subprocess in python.

We've tried to avoid the model of creating a separate deployment script for every one of our agents.

[haiyuan-eng-google]

can you tell me more about --adk_app_object flag? I want to add into our doc

[Dustan-FullStory]

Sorry it is this flag from the adk deploy agent_engine --help ouput:

  --adk_app TEXT                  Optional. Python file for defining the ADK application (default: a file named agent_engine_app.py)

In the case of this agent I am setting that flag to a value of app which is the variable name of the google.adk.apps.App object as defined in my agent.py

app = App(
    name="AIShaman",
    root_agent=root_agent,
    plugins=plugins,
)

[haiyuan-eng-google]

Great, thanks for providing the detailed context. Let me know if there is any issue on your side. Would try to address issues and unblock customer asap, thanks !

[haiyuan-eng-google]

@dvdeveloper hope your issue bigquery_agent_analytics_plugin - Agent Engine (Not Working) has also been addressed. Let me know if there is anything does not work!

[haiyuan-eng-google]

# requirements:
#   pip install "google-cloud-aiplatform[agent_engines]>=1.126.0"
# auth:
#   gcloud auth application-default login
import os
import uuid
import vertexai
# ---- Fill these in (or set as env vars) ----
PROJECT_ID = "haiyuan-dev

"
LOCATION = "us-central1"
# --- UPDATE THIS ---
AGENT_ID = "5718706211109470208"

# -------------------
AGENT_NAME = (
    f"projects/{PROJECT_ID}/locations/{LOCATION}/reasoningEngines/{AGENT_ID}"
)


def main():
  vertexai.init(project=PROJECT_ID, location=LOCATION)
  client = vertexai.Client(project=PROJECT_ID, location=LOCATION)
  try:
    print(f"Fetching agent: {AGENT_NAME}")
    # This returns an 'AgentEngine' object in your SDK version
    agent = client.agent_engines.get(name=AGENT_NAME)
    user_id = f"interactive_user_{uuid.uuid4().hex[:8]}"
    print(f"Starting interactive session with user_id: {user_id}")
    print("Type 'exit' or 'quit' to end.\n")
    while True:
      user_message = input("You: ")
      if user_message.lower() in ("exit", "quit"):
        print("Exiting.")
        break
      print("Agent: ", end="", flush=True)
      try:
        # REVERTED to standard stream_query matching your original working code
        for chunk in agent.stream_query(message=user_message, user_id=user_id):
          print(chunk, end="", flush=True)
        print("\n")
      except Exception as e:
        print(f"\nError during query: {e}")
  except Exception as e:
    print(f"\nAn error occurred while fetching the agent: {e}")


if __name__ == "__main__":
  main()

@Dustan-FullStory shared in my deployment script and test script here. I've tested the latest version package by myself and it works as expected