Skip to content

How to reprovision spectrum scale node? #699

Open
@sjpb

Description

@sjpb

Describe the bug
Having created a cluster, I deleted/recreated a vm/node/host, and ran the playbooks again. This fails, saying the node is already in the cluster. It might be from Spectrum Scale's point of view but it didn't even have the binaries. Not sure how to recover from this else I'd suggest a PR!

To Reproduce
Steps to reproduce the behavior:

  1. Install ibm-spectrum-scale-install-infra.git at v2.5.0
  2. Run playbook like this on RH8 hosts:
- name: Setup SpectrumScale
  hosts: spectrum_scale
  become: yes
  gather_facts: yes
  tags:
    - spectrum_scale
  collections:
    - ibm.spectrum_scale
  tasks:
    - include_role:
        name: ibm.spectrum_scale.core_prepare
    - include_role:
        name: ibm.spectrum_scale.core_install
    - include_role:
        name: ibm.spectrum_scale.core_configure
    - include_role:
        name: ibm.spectrum_scale.core_verify

Expected behavior
Host gets added back to the cluster.

Actual behavior

TASK [ibm.spectrum_scale.core_configure : cluster | Add new nodes to cluster] ***********************************************************************************************************************************
fatal: [compute-0]: FAILED! => {
    "changed": true,
    "cmd": [
        "/usr/lpp/mmfs/bin/mmaddnode",
        "-N",
        "/var/mmfs/tmp/NodeFile",
        "--accept"
    ],
    "delta": "0:00:08.582840",
    "end": "2023-04-19 11:32:12.819687",
    "rc": 1,
    "start": "2023-04-19 11:32:04.236847"
}

STDOUT:

mmaddnode: Node compute-5.<redacted> will be designated as possessing client license.


STDERR:

mmaddnode: Node compute-5 already belongs to the GPFS cluster.
mmaddnode: mmaddnode quitting.  None of the specified nodes are valid.
mmaddnode: Command failed. Examine previous error messages to determine cause.

Environment
Please run the following an paste your output here:

$ ansible --version
ansible [core 2.13.7]
  config file = /home/rocky/<redacted>/environments/prd/ansible.cfg
  configured module search path = ['/home/rocky/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rocky/<redacted>/venv/lib64/python3.8/site-packages/ansible
  ansible collection location = /home/rocky/<redacted>/ansible/collections
  executable location = /home/rocky/<redacted>/venv/bin/ansible
  python version = 3.8.13 (default, Nov  8 2022, 17:19:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)]
  jinja version = 3.1.2
  libyaml = True
$ python --version
Python 3.8.13

# Deployment
Spectrum_Scale_Erasure_Code-5.1.6.1-x86_64-Linux-install

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions