[backend] Problem with google.cloud.logging and set_memory_limit

[marcopellegrino-devoteam]

Environment

Platform: Vertex AI Pipelines
kfp Python library 2.14.3

When running a Kubeflow pipeline on Vertex AI, if the component has the attribute set_memory_limit, it gets 403 error when exporting the logs to Cloud Logging.
The service account correctly has the "log writer" role. I have other components in the same pipeline, exactly the same but without set_memory_limit, and they work correctly.

task = (
    example_component(
        attribute1="hello"
    ).set_memory_limit("50G").set_caching_options(
        enable_caching=False
    ).set_display_name("Test component")
)

@component(
    base_image="my-python-image",
    packages_to_install=[
        "google-cloud-logging==3.12.1",
    ],
)
def example_component(attribute1: str):
    import logging
    
    from google.cloud import logging as cloud_logging
    from google.cloud.logging.handlers import CloudLoggingHandler

    _client = cloud_logging.Client()
    _client.setup_logging(log_level=log_level)

    # Remove KFP executor duplicate log in stdout
    root_logger = logging.getLogger()
    for handler in root_logger.handlers[:]:
        # Check if the handler is a standard console stream, but NOT the Cloud handler
        if isinstance(handler, logging.StreamHandler) and not isinstance(
            handler, CloudLoggingHandler
        ):
            root_logger.removeHandler(handler)
    
   logging.warning("example")

Error logs:

2025-12-12 14:24:50.100 CET
Failed to submit 1 logs.
2025-12-12 14:24:50.100 CET
Traceback (most recent call last):
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/cloud/logging_v2/handlers/transports/background_thread.py", line 122, in _safely_commit_batch
2025-12-12 14:24:50.100 CET
batch.commit()
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/cloud/logging_v2/logger.py", line 507, in commit
2025-12-12 14:24:50.100 CET
client.logging_api.write_entries(
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/cloud/logging_v2/_gapic.py", line 167, in write_entries
2025-12-12 14:24:50.100 CET
self._gapic_api.write_log_entries(request=request)
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/cloud/logging_v2/services/logging_service_v2/client.py", line 1008, in write_log_entries
2025-12-12 14:24:50.100 CET
response = rpc(
2025-12-12 14:24:50.100 CET
^^^^
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
2025-12-12 14:24:50.100 CET
return wrapped_func(*args, **kwargs)
2025-12-12 14:24:50.100 CET
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 294, in retry_wrapped_func
2025-12-12 14:24:50.100 CET
return retry_target(
2025-12-12 14:24:50.100 CET
^^^^^^^^^^^^^
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 156, in retry_target
2025-12-12 14:24:50.100 CET
next_sleep = _retry_error_helper(
2025-12-12 14:24:50.100 CET
^^^^^^^^^^^^^^^^^^^^
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 214, in _retry_error_helper
2025-12-12 14:24:50.100 CET
raise final_exc from source_exc
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py", line 147, in retry_target
2025-12-12 14:24:50.100 CET
result = target()
2025-12-12 14:24:50.100 CET
^^^^^^^^
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/api_core/timeout.py", line 130, in func_with_timeout
2025-12-12 14:24:50.100 CET
return func(*args, **kwargs)
2025-12-12 14:24:50.100 CET
^^^^^^^^^^^^^^^^^^^^^
2025-12-12 14:24:50.100 CET
File "/workspaces/app/.venv/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
2025-12-12 14:24:50.100 CET
raise exceptions.from_grpc_error(exc) from exc
2025-12-12 14:24:50.100 CET
google.api_core.exceptions.PermissionDenied: 403 Permission 'logging.logEntries.create' denied on resource (or it may not exist). [reason: "IAM_PERMISSION_DENIED"
2025-12-12 14:24:50.100 CET
domain: "iam.googleapis.com"
2025-12-12 14:24:50.100 CET
metadata {
2025-12-12 14:24:50.100 CET
key: "permission"
2025-12-12 14:24:50.100 CET
value: "logging.logEntries.create"
2025-12-12 14:24:50.100 CET
}
2025-12-12 14:24:50.100 CET
, type_url: "type.googleapis.com/google.logging.v2.WriteLogEntriesPartialErrors"
2025-12-12 14:24:50.100 CET
value: "\nX\010\000\022T\010\007\022PPermission \'logging.logEntries.create\' denied on resource (or it may not exist)."
2025-12-12 14:24:50.100 CET
]

It has been reported before, but never corrected: #11302

[Vivekk0712]

Hi @marcopellegrino-devoteam

I'd like to work on this issue. I have GCP credits expiring soon, so I can reproduce this bug on 'actual Vertex AI Pipelines' (where it's reported to occur).

My plan:

1. Reproduce the exact error on Vertex AI with 'set_memory_limit()'
2. Document the reproduction with logs, pod specs, and service account details
3. Set up local KFP environment for deeper investigation
4. Analyze backend to identify root cause
5. Propose and test a fix

I'll update this thread with my reproduction findings within 24 hours. Could I be assigned to this issue?

Thanks!

[rishabh998186]

Hi @Vivekk0712

the set_memory_limit() was attaching pod/service-account–related annotations as dynamic attributes on the container spec but the kfp v2 compiler ignores those so they never reach the compiled ir.

try to add an official annotations field to ContainerSpecImplementation (sdk/python/kfp/dsl/structures.py) and Update pipeline_spec_builder.py to propagate these annotations into PlatformSpec.kubernetes also Refactor PipelineTask.set_memory_limit() to write to the official annotations field.
after patch compiled yaml shows the expected annotations in the PlatformSpec for steps using set_memory_limit() (previously they were missing).

I am not conform that this solve this issue so test this on real Vertex AI Pipeline .
In google cloud

Enable APIs

• Vertex AI API
• Cloud Logging API
• Cloud Storage API
  (Vertex Pipelines requires a GCS pipeline root) .

Create service account
Grant minimal roles:

• roles/vertexai.user
• roles/storage.objectAdmin (bucket)
• roles/logging.logWriter (project)
  (403 is missing logging.logEntries.create which comes from logWriter).
  ​
  Create GCS bucket
  Same region as Vertex job
  Use as PIPELINE_ROOT=gs://bucket/pipeline-root .

Run a minimal 2-step pipeline

• step A: no limits, runs google.cloud.logging.Client().setup_logging() and logs A OK
• step B: .set_memory_limit("50G") runs the exact same code and logs B OK

Print google.auth.default() in both steps to confirm identity.
if the patch works end-to-end, Step B should stop throwing 403 logging.logEntries.create.

[Vivekk0712]

Hi @rishabh998186,

Thank you so much for the detailed technical guidance! This is really helpful. 🙏

I'll follow your suggestions:

1. Patch ContainerSpecImplementation and pipeline_spec_builder.py to propagate annotations
2. Test on real Vertex AI Pipelines (since I have GCP credits expiring soon)
3. Document the reproduction and fix

I'll start working on this today and update the thread with my findings. Thanks again!

---

@marcopellegrino-devoteam - Just keeping you in the loop as the original issue reporter. I'll test this fix on GCP and report back soon.

[Vivekk0712]

@marcopellegrino-devoteam

I successfully reproduced this issue following the reproduction steps. Here are my findings.

Environment

• Platform: Vertex AI Pipelines
• KFP SDK Version: 2.15.2
• Python Version: 3.11
• Date: December 25, 2024
• Region: us-central1

Reproduction

I created a minimal 2-step pipeline as suggested by @rishabh998186:

from kfp import dsl

@dsl.component(
    base_image="python:3.11-slim",
    packages_to_install=["google-cloud-logging"]
)
def component_with_cloud_logging():
    import logging
    from google.cloud import logging as cloud_logging
    from google.cloud.logging. handlers import CloudLoggingHandler
    import google.auth
    
    credentials, project = google.auth.default()
    print(f"Running as project: {project}")
    print(f"Credentials: {type(credentials).__name__}")
    
    client = cloud_logging.Client()
    client.setup_logging(log_level=logging.INFO)
    
    root_logger = logging.getLogger()
    for handler in list(root_logger.handlers):
        if isinstance(handler, logging.StreamHandler) and not isinstance(
            handler, CloudLoggingHandler
        ):
            root_logger.removeHandler(handler)
    
    logging.warning("Testing Cloud Logging")
    print("Component completed")

@dsl.pipeline(name="test-issue-12554")
def test_pipeline():
    # Step A: no limits, runs Cloud Logging and logs "A OK"
    task_a = component_with_cloud_logging()
    task_a.set_display_name("Step A:  No Memory Limit")
    task_a.set_caching_options(enable_caching=False)
    
    # Step B: set_memory_limit("1G"), same code, logs "B OK"
    task_b = component_with_cloud_logging()
    task_b.set_memory_limit("1G")
    task_b.set_display_name("Step B: With 1G Memory Limit")
    task_b.set_caching_options(enable_caching=False)
    task_b.after(task_a)

Results

Step A (no resource limits): Completed successfully. Cloud Logging works as expected.

Step B (with set_memory_limit("1G")): Component completes but fails to write logs with 403 error:

Failed to submit 1 logs. 
google.api_core.exceptions.PermissionDenied: 403 Permission 'logging.logEntries. create' denied on resource (or it may not exist). [reason: "IAM_PERMISSION_DENIED"
domain: "iam.googleapis.com"
metadata {
  key: "permission"
  value: "logging.logEntries. create"
}

Key Finding

When Step B runs, I noticed the pod executes under a different project context:

Running as project: y5d0e482d4fc340b9-tp
Credentials:  Credentials

The project ID y5d0e482d4fc340b9-tp is not my actual project (mine is kubeflow-pipelines-482313). This suggests Vertex AI creates a restricted execution environment when resource specifications are present, and this environment lacks the necessary Cloud Logging permissions.

I inspected the compiled pipeline JSON and confirmed Step B has the resources block as expected, so the SDK is compiling correctly. This appears to be a Vertex AI platform behavior issue.

Proposed Workaround

While investigating this, I tested an alternative approach. Instead of using the google-cloud-logging client, I tried standard Python logging:

@dsl.component(base_image="python:3.11-slim")
def component_with_memory_limit():
    import logging
    
    logging.basicConfig(
        level=logging.INFO,
        format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
    )
    
    logging.info("Starting component")
    logging.warning("Component with set_memory_limit")
    print("Component completed")

Since Vertex AI automatically captures stdout and stderr to Cloud Logging, this approach bypasses the permission issue. I tested this with the same pipeline setup and it worked without 403 errors.

This workaround loses some structured logging features from the Cloud Logging SDK, but it provides a practical solution for most use cases while allowing resource limits to be used.

Next Steps

Based on @rishabh998186's suggestion about patching ContainerSpecImplementation and pipeline_spec_builder.py to propagate annotations into PlatformSpec.kubernetes, I'd like to investigate that approach further. However, the workaround above provides immediate relief for anyone blocked by this issue.

@marcopellegrino-devoteam - Hope this additional reproduction data helps. The workaround should unblock your use case while we investigate the proper fix.

[marcopellegrino-devoteam]

@Vivekk0712 thanks for the proposed workaround, actually, that is exactly what I've been currently doing myself already :)
Looking forward to have a proper fix to have the structured levels in Cloud Logging.