Skip to content

AttributeError: 'DataObject' object has no attribute '__traceback__' #1116

Description

@MajorDallas

Describe the bug

Fault objects seem to get wrapped in DataObjects, which don't have a __traceback__ attribute and causes some error handlers to crash.

This has been mentioned in #1080 (comment), and I believe it is also related to #1086.

Reproduction steps

  1. Use a framework or write an error handler that explicitly checks __traceback__. In my case, I'm using Celery.
  2. Allow the fault Exception to raise.

This is similar to my actual usage:

def start_resize(vm, user, **options):
    check_permissions(user, vm)
    result = tasks.resize_vm.s(vm, **options).apply_async()
    result.get()

# tasks.py
@celeryapp.task
def resize_vm(vm, **options):
    vc = get_vsphere_client()
    try:
        vc.change_hardware(vm, **options)
    except Exception:
        logger.exception("...")
        raise

# class VsphereClient
def change_hardware(vm, **options):
    configspec = vim.vm.ConfigSpec(**options)
    if vm.runtime.powerState == "poweredOn":
        self.power_off_vm(vm)  # raises vim.fault.InvalidPowerState if the VM is already stopped
    task = vm.ReconfigVM_Task(spec=configspec)
    WaitForTask(task, si=self.si)
    self.power_on_vm(vm)  # raises vim.fault.InvalidPowerState if the VM is already running

When the InvalidPowerState exception is raised, the resize_vm task correctly logs the exception:

pyVmomi.VmomiSupport.vim.fault.InvalidPowerState: (vim.fault.InvalidPowerState) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'The attempted operation cannot be performed in the current state (Powered on).',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   requestedState = 'poweredOff',
   existingState = 'poweredOn'
}

The problem begins after the call to logger.exception, when the exception is re-raised:

< the InvalidPowerState traceback omitted for brevity >

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 605, in trace_task
    I, R, state, retval = on_error(task_request, exc)
                          ~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 421, in on_error
    R = I.handle_error_state(
        task, request, eager=eager, call_errbacks=call_errbacks,
    )
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 199, in handle_error_state
    return {
           ~
        RETRY: self.handle_retry,
        ~~~~~~~~~~~~~~~~~~~~~~~~~
        FAILURE: self.handle_failure,
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    }[self.state](task, req,
    ~~~~~~~~~~~~~^^^^^^^^^^^
                  store_errors=store_errors,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
                  call_errbacks=call_errbacks)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 249, in handle_failure
    if exc.__traceback__ is None:
       ^^^^^^^^^^^^^^^^^
AttributeError: 'DataObject' object has no attribute '__traceback__'

InvalidPowerState happens to be the one that happened today, but I've run into this with other VimFault types. In #1080, it was a NotFound fault.

Expected behavior

Re-raising a vim.fault.VimFault should not pass a DataObject up to the next exception handler. Or, if that can't be avoided, a DataObject that wraps a VimFault should proxy the __traceback__ attribute to the wrapped fault object's __traceback__.

Additional context

While I am running into this with Celery, this could be expected to occur with any exception handler that accesses __traceback__ on exception objects.

I mentioned this same problem in #1080, which was primarily focused on static typing. However, this is a runtime erasure of exception information.

#1086 also points out that VimFault itself is not an exception type. I mentioned the same concern in #1080. It appears Exception doesn't get added to the MRO for faults until runtime, which is certainly a pain for type checkers, but, more importantly, it also seems that there are circumstances where the VimFault is wrapped in a DataObject--which is definitely not a BaseException even at runtime. That would make it impossible to catch, in addition to erasing the __traceback__.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions