(nucleus): AWS_IOT_THING_NAME variable empty in environment of my components

[Dizzzmas]

Describe the bug
We use fleet provisioning to roll greengrass onto our devices.
In the last couple weeks we've encountered the below issue on 2 different newly installed devices:

Our deployed greengrass components have empty AWS_IOT_THING_NAME env variable

When running cat /proc/<process_id>/environ | tr '\0' '\n'
where <process_id> belongs to the greengrass component, running on device via greengrass deployment, we see the following picture:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANGUAGE=en_US.UTF-8
INVOCATION_ID=f832fb4ded1b4ee7a25e491ee6c95a1d
AWS_REGION=eu-west-2

AWS_IOT_THING_NAME= # EMPTY!!!

PIDFILE=/greengrass/v2/alts/loader.pid
AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH=/greengrass/v2/ipc.socket
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
AWS_DEFAULT_REGION=eu-west-2
LC_MESSAGES=en_US.UTF-8
AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT=/greengrass/v2/ipc.socket
AWS_GG_FIPS_MODE=false
LANG=en_US.UTF-8
SVCUID=BEZ2SJJXUUPYUAHB
JOURNAL_STREAM=8:9713
GG_ROOT_CA_PATH=/greengrass/v2/claim-certs/claim.root.pem
GGC_VERSION=2.12.6
AWS_CONTAINER_AUTHORIZATION_TOKEN=BEZ2SJJXUUPYUAHB
SYSTEMD_EXEC_PID=1591
AWS_USE_FIPS_ENDPOINT=false
PWD=/greengrass/v2/alts/current/distro/bin
HOME=/root
AWS_CONTAINER_CREDENTIALS_FULL_URI=http://localhost:45847/2016-11-01/credentialprovider/

For some reason AWS_IOT_THING_NAME is empty, and this is not solved by a reboot.

I can see the Thing having correct name inside IoT Core and /greengrass/v2/config/effectiveConfig.yaml has the correct thingName on the device

Expected behavior
Every greengrass component always has AWS_IOT_THING_NAME set at runtime

Actual behavior
Our greengrass components have empty AWS_IOT_THING_NAME

Environment

• OS: Ubuntu 22.04
• JDK version: 11.0.23+9-post-Ubuntu-1ubuntu122.04.1
• Nucleus version: 2.12.6

[MikeDombo]

Where do you see the thing name in the effective config? It must be

system:
  thingName: <your thing name>

[Dizzzmas]

Where do you see the thing name in the effective config? It must be

system:
  thingName: <your thing name>

seems correct

# /greengrass/v2/config/effectiveConfig.yaml
---
system:
  certificateFilePath: "/greengrass/v2/thingCert.crt"
  privateKeyPath: "/greengrass/v2/privKey.key"
  rootCaPath: "/greengrass/v2/claim-certs/claim.root.pem"
  rootpath: "/greengrass/v2"
  thingName: "Recorder_FRTH"
services:
.....

[MikeDombo]

Is your application running under sudo? Sudo does not pass through environment variables by default.

[Dizzzmas]

Is your application running under sudo? Sudo does not pass through environment variables by default.

we're using RequiresPrivilege: true in component recipe
so it should be running as a superuser

but it would be strange for other AWS_* variables to propagate and for AWS_IOT_THING_NAME not to

Here's the full component recipe:

---
RecipeFormatVersion: "2020-01-25"
ComponentName: "{COMPONENT_NAME}"
ComponentVersion: "{COMPONENT_VERSION}"
ComponentDescription: "Recorder daemon responsible for orchestrating recording, mixing, etc."
ComponentPublisher: "{COMPONENT_AUTHOR}"

ComponentConfiguration:
  DefaultConfiguration:
    accessControl:
      aws.greengrass.ipc.mqttproxy:
        com.setmixer.recorder:mqttproxy:1:
          policyDescription: Allow publishing to IoT Core topics
          operations:
            - "*"
          resources:
            - "*"
      aws.greengrass.ShadowManager:
        com.setmixer.recorder:shadow:1:
          policyDescription: "Allow access to local shadows"
          operations:
            - "aws.greengrass#GetThingShadow"
            - "aws.greengrass#UpdateThingShadow"
            - "aws.greengrass#DeleteThingShadow"
          resources:
            - "*"
        com.setmixer.recorder:shadow:2:
          policyDescription: "Allow listing thing shadows"
          operations:
            - "aws.greengrass#ListNamedShadowsForThing"
          resources:
            - "*"

      aws.greengrass.ipc.pubsub:
        com.setmixer.recorder:pubsub:1:
          policyDescription: "Allow access to local shadow pub/sub topics"
          operations:
            - "aws.greengrass#SubscribeToTopic"
            - "aws.greengrass#PublishToTopic"
          resources:
            - "*"

ComponentDependencies:
  com.setmixer.assets:
    VersionRequirement: "*"
    DependencyType: SOFT
  org.python.cpython:
    VersionRequirement: "~3.10.0"
    DependencyType: SOFT

Manifests:
  - Platform:
      os: all
    Artifacts:
      - URI: "s3://BUCKET_NAME/COMPONENT_NAME/COMPONENT_VERSION/com.setmixer.recorder.zip"
        Unarchive: ZIP
    Lifecycle:
      Run:
        Script: "chmod a+x {artifacts:decompressedPath}/com.setmixer.recorder/setmixerd && {artifacts:decompressedPath}/com.setmixer.recorder/setmixerd"
        RequiresPrivilege: true
        SetEnv:
          SMX_SHARE_PATH: "{com.setmixer.assets:artifacts:decompressedPath}/com.setmixer.assets"

and here's its place in deployment manifest:

    "com.setmixer.recorder": {
      "componentVersion": "0.65.1",
      "configurationUpdate": {
        "reset": [
          ""
        ]
      }
    },

[Dizzzmas]

wonder if it could be caused by our fleet provisioning process being a bit weird and hacky

greengrass is expected to start only after an operator on-site unboxes the device and specifies its "code" - essentially the thing name

This is our greengrass.service

# greengrass.service
[Unit]
Description=Greengrass Core
After=network-online.target
After=wait-for-first-login.service  # this is responsible for handling user's thing name input
Wants=wait-for-first-login.service

[Service]
ExecStartPre=/bin/sh -c "/greengrass/v2/config/replace_thing_id.sh" 
....

and this is our ExecStartPre script that's supposed to run before fleet provisioning happens:

#!/bin/bash

sleep 20

hostname=$(cat /etc/hostname)

# Assumes hostname is always in the format recorder-TARGET
location_code=${hostname#recorder-}

if [ -n "$location_code" ]  && [ "$location_code" != "null" ] && [ "$location_code" != "0" ]; then
sed -i "s/<unique>/$location_code/g" "/greengrass/v2/config/config.yaml"
sed -i "s/<unique>/$location_code/g" "/greengrass/v2/config/config.tlog"
sed -i "s/<unique>/$location_code/g" "/greengrass/v2/config/effectiveConfig.yaml"
sed -i "s/<unique>/$location_code/g" "/greengrass/v2/config/effectiveConfig.yaml~"
fi

[rawalexe]

What's in your config.tlog does it actually has updated values? There might a race condition on your script and the nucleus transactions. You aren't supposed to modify values as such, actual config is stored in config.tlog not in effectiveConfig. Can you not just provide template parameters as under the fleet-prov config instead of this hack?

[Dizzzmas]

What's in your config.tlog does it actually has updated values? There might a race condition on your script and the nucleus transactions. You aren't supposed to modify values as such, actual config is stored in config.tlog not in effectiveConfig. Can you not just provide template parameters as under the fleet-prov config instead of this hack?

seems like config.tlog has a record for thingName changing

$ grep thingName /greengrass/v2/config/config.tlog

{"TS":1766336646698,"TP":["system","thingName"],"W":"changed","V":"Recorder_FRTH"}

Can you not just provide template parameters as under the fleet-prov config instead of this hack?

can you please point to some sort of reference?
reason for this hack was that thingName is not known until the user inputs it

[rawalexe]

Fleet provisioning as it is today won't fit your use case. We would need to subscribe to the topic and restart nucleus with the change. This is a feature request and I can discuss with the team on how to prioritize it.

ideal path

Greengrass starts with initial config --> your custom script runs --> fleet provisioning detects the config change --> Fleet prov re-inits GG's config.tlog

As a quick fix maybe you should run your custom script update from GG runs. So try adding it to GG's ExecPre. This can also lead to correct behavior

[rawalexe]

If you can provide it early providing the seed for thingName should be provided under templateParams as the following in the current scope

 configuration:
      iotDataEndpoint: "aaaaaaaaaaaaaa-ats.iot.us-east-1.amazonaws.com" #[Modify here]
      iotCredEndpoint: "cccccccccccccc.credentials.iot.us-east-1.amazonaws.com" #[Modify here]
      .
      .
      templateParams: '{"SerialNumber": "a2_b9_d2_5a_fd_f9"}' #[Modify here]