Cannot relate jenkins-k8s-agent

[medicalwei]

Bug Description

We have problem relating jenkins-k8s-agent to jenkins-k8s, juju debug-log suggests a key error to state.agent_relation_meta. The deployment was initialized using terraform, but manually removing and adding agent relation does not resolve neither.

To Reproduce

Deploy with the following Terraform config with juju plugin

locals {
  juju_model_name = "some_model_name"
}

data "juju_model" "service_model" {
  name = local.juju_model_name
}

resource "juju_application" "jenkins" {
  name = "jenkins"

  model = data.juju_model.service_model.name

  charm {
    name     = "jenkins-k8s"
    channel  = "latest/edge"
    revision = 177
  }

  config = {
    allowed-plugins = "" # allow all plugins
  }

  storage_directives = {
    jenkins-home = "20G"
  }

  units = 1
}

resource "juju_application" "jenkins-agent-k8s" {
  name = "jenkins-agent-k8s"

  model = data.juju_model.service_model.name

  charm {
    name     = "jenkins-agent-k8s"
    channel  = "latest/edge"
    revision = 36
  }

  constraints = "arch=amd64 root-disk=153600M"

  units = 5
}

resource "juju_application" "nginx-ingress-integrator" {
  name = "nginx-ingress-integrator"

  model = data.juju_model.service_model.name

  charm {
    name     = "nginx-ingress-integrator"
    channel  = "latest/stable"
    revision = 203
  }

  config = {
    path-routes = "/"
    service-hostname = "jenkins.example.com"
  }

  units = 1 
  trust = true
}

resource "juju_integration" "jenkins_jenkins-agent-k8s" {
  model = data.juju_model.service_model.name

  application {
    name     = juju_application.jenkins.name
    endpoint = "agent"
  }

  application {
    name     = juju_application.jenkins-agent-k8s.name
    endpoint = "agent"
  }
}

resource "juju_integration" "jenkins_ingress" {
  model = data.juju_model.service_model.name

  application {
    name     = juju_application.jenkins.name
    endpoint = "ingress"
  }

  application {
    name     = juju_application.nginx-ingress-integrator.name
    endpoint = "ingress"
  }
}

Environment

Juju Controller version 3.6.8
jenkins-k8s r177
jenkins-agent-k8s r36
nginx-ingress-integrator r203

Relevant log output

unit-jenkins-0: 01:55:24 DEBUG unit.jenkins/0.juju-log agent:17: ops 3.2.0 up and running.
unit-jenkins-0: 01:55:24 DEBUG unit.jenkins/0.juju-log agent:17: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-jenkins-0/charm/src/prometheus_alert_rules: directory does not exist
unit-jenkins-0: 01:55:24 DEBUG unit.jenkins/0.juju-log agent:17: Emitting Juju event agent_relation_joined.
unit-jenkins-0: 01:55:24 DEBUG unit.jenkins/0.agent-relation-joined ServiceInfo(name='jenkins', startup=ServiceStartup.ENABLED, current=ServiceStatus.ACTIVE)
unit-jenkins-0: 01:55:24 ERROR unit.jenkins/0.juju-log agent:17: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-jenkins-0/charm/./src/charm.py", line 211, in <module>
    ops.main.main(JenkinsK8sOperatorCharm)
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/__init__.py", line 359, in main
    return _legacy_main.main(
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/main.py", line 39, in main
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/_main.py", line 504, in main
    manager.run()
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/_main.py", line 488, in run
    self._emit()
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/_main.py", line 423, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/_main.py", line 467, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-jenkins-0/charm/venv/ops/framework.py", line 1036, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-jenkins-0/charm/src/agent.py", line 166, in _on_agent_relation_joined
    agent_meta = agent_relation_meta[typing.cast(ops.Unit, event.unit).name]
KeyError: 'jenkins-agent-k8s/0'
unit-jenkins-0: 01:55:25 ERROR juju.worker.uniter.operation hook "agent-relation-joined" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response

[yanksyoon]

Hi, thanks for the report! Let me create a ticket to follow up :)

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/ISD-4183.

This message was autogenerated

[yanksyoon]

Hi @medicalwei ! Would you mind trying with the latest revision please? It's been fixed with #265 :)

[medicalwei]

now it does not show up as an error, however jenkins-agent-k8s are kept waiting:

$ juju status
Model                 Controller                         Cloud/Region              Version  SLA          Timestamp
prod-oem-jenkins-ps6  juju-controller-36-production-ps6  k8s-prod-general/default  3.6.8    unsupported  03:54:44Z

App                             Version  Status   Scale  Charm                     Channel        Rev  Address        Exposed  Message
jenkins                         2.516.2  active       1  jenkins-k8s               latest/edge    181  xxxx   no       
jenkins-agent-k8s                        waiting      5  jenkins-agent-k8s         latest/edge     37  xxxx   no       Waiting for complete relation data.
nginx-ingress-integrator        24.2.0   active       1  nginx-ingress-integrator  latest/stable  203  xxxx   no       Ingress IP(s): xxxx
nginx-ingress-integrator-agent  24.2.0   active       1  nginx-ingress-integrator  latest/stable  203  xxxx   no       Ingress IP(s): xxxx

Unit                               Workload  Agent  Address         Ports  Message
jenkins-agent-k8s/0                waiting   idle   192.168.98.210         Waiting for complete relation data.
jenkins-agent-k8s/1                waiting   idle   192.168.96.16          Waiting for complete relation data.
jenkins-agent-k8s/2                waiting   idle   192.168.97.244         Waiting for complete relation data.
jenkins-agent-k8s/3                waiting   idle   192.168.97.120         Waiting for complete relation data.
jenkins-agent-k8s/4*               waiting   idle   192.168.96.147         Waiting for complete relation data.
jenkins/0*                         active    idle   192.168.97.252         
nginx-ingress-integrator-agent/0*  active    idle   192.168.97.197         Ingress IP(s): xxxx
nginx-ingress-integrator/0*        active    idle   192.168.98.209         Ingress IP(s): xxxx

[yanksyoon]

@medicalwei Hello! Thanks for following up on the issue - may I know if the output log above was done by upgrade-charm or a redeployment?
The upgrade-charm had a bug where it didn't quite reconcile the agents (this charm is not yet using the recommended reconcile patter). This has been addressed by: #272.
In any case, upgrade or redeploy should both work at the moment, would you mind trying to upgrade/redeploy again please?
Sorry about the hassle.

[medicalwei]

During my attempt the machines are all redeployed (terraform destroy or equivalent) then rebuilt. I can successfully deploy now, but is after changing the jenkins-k8s application name from jenkins to jenkins-k8s. I am not sure why the name of the deployed application can affect relations.

[yanksyoon]

@medicalwei , I've created a separate issue to handle this, thanks again for your report :)
I'll close this one.