ERROR Exceeded maximum object depth. This may have been caused by excessive role recursion. maximum recursion depth exceeded

[sarahbx]

While attempting a deploy, I have run into this error:

ERROR! Exceeded maximum object depth. This may have been caused by excessive role recursion. maximum recursion depth exceeded

There is an issue with the cluster, causing it to continue hitting the rescue block, but I am opening this issue because the process needs to be changed from a recursive call to an iterative call to avoid hitting the above error and masking the underlying problem.

This recursive behavior is seen in at least two places:

jetlag/ansible/roles/install-cluster/tasks/check-cluster-install.yml

Line 152 in 7143305

include_tasks: check-cluster-install.yml

jetlag/ansible/roles/ocp-scale-out-csr/tasks/check_nodes_joined.yml

Line 58 in 7143305

include_tasks: check_nodes_joined.yml

When I have an opportunity, I will look at a possible fix, but this would also be a good issue for a first time contributor.

[sarahbx]

Found the process was not pausing at all due to being run with the output redirected to null.

INFO| Pausing for 600 seconds
INFO| fatal: [<HOST>]: FAILED! => {
    "msg": "[Errno 25] Inappropriate ioctl for device"
}

I am running jetlag within a python subprocess that redirects to null.
ANSIBLE_LOG_PATH=... ansible-playbook {ARGS} >/dev/null 2>&1

A pause issue was opened on ansible for same error, also when not used in an interactive terminal.
ansible/ansible#85703

[akrzos]

@sarahbx Did you end up solving your issue? Was there a specific method to prevent this issue if you end up calling an ansible-playbook via a python subprocess that directs the output to null?

[sarahbx]

Hi @akrzos , I had two scenarios where this was used. In subprocess, as well as using paramiko.

Both have PTY work-arounds that can be applied.

Subprocess:

# utilities/shell.py
from __future__ import annotations

import errno
import logging
import os
import pty
import sys
import shlex
import subprocess
from typing import Optional


LOGGER = logging.getLogger(__name__)


def open_pty() -> tuple[int, int]:
    """Allocate a pseudo-terminal pair.

    Used to provide a TTY to subprocesses that require one (e.g., Ansible's
    pause module). This works around upstream Ansible bug ansible/ansible#85703.

    Returns:
        Tuple of (parent_fd, child_fd). The child_fd is passed as stdin to the
        subprocess; the parent_fd is closed after fork to send EOF.
    """
    parent_fd, child_fd = pty.openpty()
    LOGGER.debug(f"Allocated PTY: parent_fd={parent_fd}, child_fd={child_fd}")
    return parent_fd, child_fd


def close_fd(fd: int, label: str) -> None:
    """Close a file descriptor with logging.

    Args:
        fd: File descriptor to close.
        label: Human-readable label for debug logging (e.g., "pty_parent").

    Raises:
        OSError: If close fails for a reason other than the FD already being
            closed (EBADF). An already-closed FD is logged as a warning.
    """
    try:
        os.close(fd)
        LOGGER.debug(f"Closed FD {fd} ({label})")
    except OSError as err:
        if err.errno == errno.EBADF:
            LOGGER.warning(f"FD {fd} ({label}) was already closed")
        else:
            raise


def bash(
    command: str,
    timeout: Optional[int] = None,
    cwd: Optional[str] = None,
    env: Optional[dict] = None,
    capture: bool = False,
    sensitive: bool = False,
    allocate_pty: bool = False,
) -> tuple[int, str, str]:
    """
    Run a command in bash.

    Args:
        command: The command to execute
        timeout: Optional timeout in seconds
        cwd: Working directory for the command
        env: Environment variables for the command
        capture: If True, capture and return stdout/stderr as strings
        sensitive: If True, do not log the command (for commands containing secrets)
        allocate_pty: If True, allocate a PTY for stdin so the child process
            sees a TTY. Used for Ansible playbooks that require a TTY
            (workaround for ansible/ansible#85703).

    Returns:
        Tuple of (return_code, stdout, stderr). stdout/stderr are empty strings if not capturing.
    """
    if sensitive:
        LOGGER.info("Running: <sensitive command redacted>")
    else:
        LOGGER.info(f"Running: {command!r}")

    parent_fd = None
    child_fd = None
    stdin_arg = None

    if allocate_pty:
        parent_fd, child_fd = open_pty()
        stdin_arg = child_fd

    try:
        args = shlex.split(f"/usr/bin/env bash -e -c {shlex.quote(command)}")
        with subprocess.Popen(
            args=args,
            stdin=stdin_arg,
            stdout=subprocess.PIPE if capture else sys.stdout,
            stderr=subprocess.PIPE if capture else sys.stderr,
            cwd=cwd,
            env=env,
        ) as proc:
            # Close child FD in parent after fork — child process inherits it.
            # Close parent FD to send EOF on child stdin.
            if child_fd is not None:
                close_fd(fd=child_fd, label="pty_child")
                child_fd = None
            if parent_fd is not None:
                close_fd(fd=parent_fd, label="pty_parent")
                parent_fd = None

            proc.wait(timeout=timeout)
            if capture and proc.stdout is not None and proc.stderr is not None:
                stdout = proc.stdout.read().decode("utf-8", errors="replace")
                stderr = proc.stderr.read().decode("utf-8", errors="replace")
                return proc.returncode, stdout, stderr
            else:
                return proc.returncode, "", ""
    finally:
        # Ensure FDs are closed on any exception path
        if child_fd is not None:
            close_fd(child_fd, "pty_child")
        if parent_fd is not None:
            close_fd(parent_fd, "pty_parent")

Paramiko

https://docs.paramiko.org/en/latest/api/client.html#paramiko.client.SSHClient.exec_command

self._client: paramiko.SSHClient
...
stdin, stdout, stderr = self._client.exec_command(command=command, timeout=timeout, get_pty=get_pty)

[akrzos]

Got it, so with those workarounds it sounds like you are all set then and we can close this issue out now right?

[sarahbx]

It resolved the PTY issue, but I would suggest keeping this Issue open until the recursion issue is resolved.

[akrzos]

Sure, the recursion limit is hit generally when there is a hardware issue or mis-configuration that prevents successful install of a cluster. The original method to detect fully installed clusters was simply just a looped task however over time I found that we could solve some issues in the lab during the installation loop. For example if an incorrect virtual media boot order is on a machine, we can detect it and eject the disk allowing the install to complete without intervention. Hence we moved to the recursive loop such that remediation actions could be made. I'm not so such the same can be accomplished without recursion.