(Greengrass): Failed to write certificate with custom install directory and fleet provisioning

[mpichette]

Describe the bug
I'm on he Scarthgap branch and trying to install Greengrass in a data partition. I'm getting errors on start when it's trying to provision for the first time. I have tried with and without specifying the certPath and keyPath but it's the same issue in both cases and the error message still refers to the root path so it looks like it's trying to write to that location no matter what.

Any idea how to get fleet provisioning working with a custom install directory (I don't care if the key or cert are installed in the root path).

To Reproduce
My effectiveConfig.yaml

(yaml)
system:
  certificateFilePath: ""
  privateKeyPath: ""
  rootCaPath: ""
  rootpath: "/data/opt/dk/greengrass/v2"
  thingName: ""
services:
  aws.greengrass.FleetProvisioningByClaim:
    configuration:
      awsRegion: "us-east-1"
      certPath: "/opt/dk/greengrass/v2/keys/thingCert.crt"
      claimCertificatePath: "/opt/dk/greengrass/v2/claim-certs/claim.cert.pem"
      claimCertificatePrivateKeyPath: "/opt/dk/greengrass/v2/claim-certs/claim.pkey.pem"
      iotCredentialEndpoint: "XXXXX.credentials.iot.us-east-1.amazonaws.com"
      iotDataEndpoint: "XXX-ats.iot.us-east-1.amazonaws.com"
      iotRoleAlias: "XXXX-alias"
      keyPath: "/opt/dk/greengrass/v2/keys/privKey.key"
      provisioningTemplate: "XXXXX-template"
      rootCaPath: "/opt/dk/greengrass/v2/claim-certs/claim.root.pem"
      rootPath: "/opt/dk/greengrass/v2"
      templateParameters:
        MacAddress: "XX:XX:XX:XX:XX:XX"
        SerialNumber: "12345"
        ThingName: "XX-XX-XX-XX-XX-XX"
  aws.greengrass.Nucleus:
    componentType: "NUCLEUS"
    configuration:
      awsRegion: "us-east-1"
      componentStoreMaxSizeBytes: 10000000000
      deploymentPollingFrequencySeconds: 15
      fipsMode: "false"
      iotCredEndpoint: ""
      iotDataEndpoint: ""
      jvmOptions: "-Dlog.store=FILE"
      logging: {}
      mqtt:
        spooler: {}
      platformOverride: {}
      runWithDefault:
        posixUser: "ggc_user:ggc_group"
      telemetry: {}
    dependencies: []
    lifecycle:
      bootstrap:
        requiresPrivilege: true
        script: "\nset -eu\nKERNEL_ROOT=\"/data/opt/dk/greengrass/v2\"\nUNPACK_DIR=\"\
          /data/opt/dk/greengrass/v2/packages/artifacts-unarchived/aws.greengrass.Nucleus/2.16.0/aws.greengrass.nucleus\"\
          \nrm -r \"$KERNEL_ROOT\"/alts/current/*\necho \"-Dlog.store=FILE\" > \"\
          $KERNEL_ROOT/alts/current/launch.params\"\nln -sf \"$UNPACK_DIR\" \"$KERNEL_ROOT/alts/current/distro\"\
          \nexit 100"
    version: "2.16.0"
  DeploymentService:
    dependencies: []
    version: "0.0.0"
  FleetStatusService:
    dependencies: []
  main:
    dependencies:
    - "aws.greengrass.Nucleus"
    lifecycle: {}
  TelemetryAgent:
    dependencies: []
    runtime:
      lastPeriodicAggregationMetricsTime: 1772568041217
      lastPeriodicPublishMetricsTime: 1772568041218
    version: "0.0.0"
  UpdateSystemPolicyService:
    dependencies: []
    version: "0.0.0"

Expected behavior
Would expect provisioning to complete and have private key and cert written to disk in the correct location.

Actual behavior
Tell us what actually happened.

Environment

• OS: Yocto 6.6.63-v8 kernel version - Scarthgap
• JDK version: openjdk version "11.0.30" 2026-01-20 LTS
• Nucleus version: 2.16.0

Additional context
Associated issue: aws4embeddedlinux/meta-aws#15192

Error logs:

2026-03-03T20:00:38.542Z [INFO] (main) com.aws.greengrass.util.platforms.unix.linux.LinuxSystemResourceController: Detected cgroup v2 system. {}
2026-03-03T20:00:38.564Z [INFO] (main) com.aws.greengrass.util.platforms.Platform: Getting platform instance com.aws.greengrass.util.platforms.unix.linux.LinuxPlatform.. {}
2026-03-03T20:00:38.623Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: No ongoing deployment detected. Proceed as default. {}
2026-03-03T20:00:38.639Z [WARN] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Transaction log /data/opt/dk/greengrass/v2/config/config.tlog is invalid and no usable backup transaction log exists. Either an initial Nucleus setup is ongoing or all config tlogs were corrupted. {}
2026-03-03T20:00:38.640Z [INFO] (main) com.aws.greengrass.config.Configuration: config-loading. Read configuration from a file path. {path=/data/opt/dk/greengrass/v2/config/config.yaml}
2026-03-03T20:00:39.094Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/data/opt/dk/greengrass/v2/config/effectiveConfig.yaml}
2026-03-03T20:00:39.154Z [INFO] (main) com.aws.greengrass.security.SecurityService: Register crypto key service provider. {keyType=file}
2026-03-03T20:00:39.157Z [INFO] (main) com.aws.greengrass.security.SecurityService: Register MQTT connection security provider. {keyType=file}
2026-03-03T20:00:39.251Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Successfully setup Nucleus launch parameters. {}
2026-03-03T20:00:39.669Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Nucleus lifecycle has been initialized successfully. {}
2026-03-03T20:00:39.865Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: system-start. Launch Nucleus. {configPath=/data/opt/dk/greengrass/v2/config, rootPath=/data/opt/dk/greengrass/v2, version=2.16.0}
2026-03-03T20:00:40.416Z [INFO] (main) software.amazon.awssdk.eventstreamrpc.RpcServer: IpcServer started.... {}
2026-03-03T20:00:40.585Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: serviceImplementors: {UpdateSystemPolicyService:class com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService...}. {}
2026-03-03T20:00:40.642Z [INFO] (main) com.aws.greengrass.mqttclient.spool.Spool: Spooler has been configured. {maxSizeInBytes=2621440, storageType=Memory, keepQos0WhenOffline=false}
2026-03-03T20:00:40.645Z [INFO] (main) com.aws.greengrass.mqttclient.spool.Spool: Memory Spooler has been set up. {}
2026-03-03T20:00:40.686Z [INFO] (pool-3-thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Running provisioning plugin: aws.greengrass.FleetProvisioningByClaim. {}
2026-03-03T20:00:40.691Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=aws.greengrass.Nucleus}
2026-03-03T20:00:40.791Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=main}
2026-03-03T20:00:40.840Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=UpdateSystemPolicyService}
2026-03-03T20:00:40.847Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=DeploymentService}
2026-03-03T20:00:41.020Z [WARN] (main) com.aws.greengrass.status.FleetStatusService: Device not configured to talk to AWS IoT cloud. FleetStatusService is offline. {errorMessage=[thingName cannot be empty, certificateFilePath cannot be empty, privateKeyPath cannot be empty, rootCaPath cannot be empty, iotDataEndpoint cannot be empty, iotCredEndpoint cannot be empty], serviceName=FleetStatusService, currentState=NEW}
2026-03-03T20:00:41.021Z [WARN] (main) com.aws.greengrass.status.FleetStatusService: Device not configured to talk to AWS IoT cloud. FleetStatusService is offline. {errorMessage=[thingName cannot be empty, certificateFilePath cannot be empty, privateKeyPath cannot be empty, rootCaPath cannot be empty, iotDataEndpoint cannot be empty, iotCredEndpoint cannot be empty], serviceName=FleetStatusService, currentState=NEW}
2026-03-03T20:00:41.033Z [WARN] (main) com.aws.greengrass.deployment.IotJobsHelper: Device not configured to talk to AWS Iot cloud. IOT job deployment is offline. {errorMessage=[thingName cannot be empty, certificateFilePath cannot be empty, privateKeyPath cannot be empty, rootCaPath cannot be empty, iotDataEndpoint cannot be empty, iotCredEndpoint cannot be empty]}
2026-03-03T20:00:41.048Z [WARN] (main) com.aws.greengrass.deployment.ShadowDeploymentListener: Device not configured to talk to AWS Iot cloud. Single device deployment is offline. {}
2026-03-03T20:00:41.051Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=TelemetryAgent}
2026-03-03T20:00:41.214Z [INFO] (pool-3-thread-1) com.aws.greengrass.IotIdentityHelper: Subscribed to CreateKeysAndCertificateAccepted. {}
2026-03-03T20:00:41.297Z [INFO] (pool-3-thread-1) com.aws.greengrass.IotIdentityHelper: Subscribed to CreateKeysAndCertificateRejected. {}
2026-03-03T20:00:41.304Z [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/data/opt/dk/greengrass/v2/config/effectiveConfig.yaml}
2026-03-03T20:00:41.305Z [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: system-start. {main=services.main:NEW}
2026-03-03T20:00:41.328Z [WARN] (main) com.aws.greengrass.status.FleetStatusService: Status won't be published until Nucleus is configured online. {trigger=NUCLEUS_LAUNCH, serviceName=FleetStatusService, currentState=NEW}
2026-03-03T20:00:41.330Z [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=NEW, newState=INSTALLED}
2026-03-03T20:00:41.330Z [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=NEW, newState=INSTALLED}
2026-03-03T20:00:41.331Z [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=NEW, newState=INSTALLED}
2026-03-03T20:00:41.331Z [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=NEW, newState=INSTALLED}
2026-03-03T20:00:41.332Z [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=NEW, newState=INSTALLED}
2026-03-03T20:00:41.332Z [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=NEW, newState=INSTALLED}
2026-03-03T20:00:41.343Z [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=INSTALLED, newState=STARTING}
2026-03-03T20:00:41.344Z [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=INSTALLED, newState=STARTING}
2026-03-03T20:00:41.345Z [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=INSTALLED, newState=STARTING}
2026-03-03T20:00:41.348Z [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=INSTALLED, newState=STARTING}
2026-03-03T20:00:41.348Z [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=INSTALLED, newState=STARTING}
2026-03-03T20:00:41.361Z [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=STARTING, newState=RUNNING}
2026-03-03T20:00:41.361Z [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=STARTING, newState=RUNNING}
2026-03-03T20:00:41.364Z [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=STARTING, newState=RUNNING}
2026-03-03T20:00:41.367Z [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=STARTING, newState=RUNNING}
2026-03-03T20:00:41.368Z [INFO] (pool-3-thread-7) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-finished. Nothing done. {serviceName=aws.greengrass.Nucleus, currentState=STARTING}
2026-03-03T20:00:41.369Z [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=STARTING, newState=FINISHED}
2026-03-03T20:00:41.372Z [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=INSTALLED, newState=STARTING}
2026-03-03T20:00:41.374Z [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=FINISHED, newState=STOPPING}
2026-03-03T20:00:41.378Z [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=STARTING, newState=FINISHED}
2026-03-03T20:00:41.378Z [INFO] (pool-3-thread-7) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
2026-03-03T20:00:41.378Z [INFO] (pool-3-thread-9) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-finished. Nothing done. {serviceName=main, currentState=STARTING}
2026-03-03T20:00:41.380Z [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=FINISHED, newState=STOPPING}
2026-03-03T20:00:41.381Z [INFO] (pool-3-thread-9) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=main, currentState=STOPPING}
2026-03-03T20:00:41.382Z [INFO] (pool-3-thread-7) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
2026-03-03T20:00:41.383Z [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING, newState=FINISHED}
2026-03-03T20:00:41.383Z [INFO] (pool-3-thread-9) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=main, currentState=STOPPING}
2026-03-03T20:00:41.384Z [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=STOPPING, newState=FINISHED}
2026-03-03T20:00:41.422Z [INFO] (pool-3-thread-1) com.aws.greengrass.IotIdentityHelper: Published to CreateKeysAndCertificate. {}
2026-03-03T20:00:41.696Z [ERROR] (pool-3-thread-1) com.aws.greengrass.FleetProvisioningByClaimPlugin: Caught exception while writing certificate to file. {}
java.nio.file.FileAlreadyExistsException: /opt/dk/greengrass/v2
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
	at java.base/java.nio.file.Files.createDirectory(Files.java:690)
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
	at java.base/java.nio.file.Files.createDirectories(Files.java:743)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.createFile(FleetProvisioningByClaimPlugin.java:381)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.writeFile(FleetProvisioningByClaimPlugin.java:374)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.writeCertificate(FleetProvisioningByClaimPlugin.java:365)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.writeCertificate(FleetProvisioningByClaimPlugin.java:360)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.updateIdentityConfiguration(FleetProvisioningByClaimPlugin.java:176)
	at com.aws.greengrass.lifecyclemanager.KernelLifecycle.lambda$executeProvisioningPlugin$1(KernelLifecycle.java:225)
	at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:77)
	at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:45)
	at com.aws.greengrass.lifecyclemanager.KernelLifecycle.lambda$executeProvisioningPlugin$2(KernelLifecycle.java:224)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

2026-03-03T20:00:41.704Z [ERROR] (pool-3-thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Caught exception while running provisioning plugin. Moving on to run Greengrass without provisioning. {}
com.aws.greengrass.DeviceProvisioningRuntimeException: Failed to write certificate
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.writeCertificate(FleetProvisioningByClaimPlugin.java:369)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.writeCertificate(FleetProvisioningByClaimPlugin.java:360)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.updateIdentityConfiguration(FleetProvisioningByClaimPlugin.java:176)
	at com.aws.greengrass.lifecyclemanager.KernelLifecycle.lambda$executeProvisioningPlugin$1(KernelLifecycle.java:225)
	at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:77)
	at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:45)
	at com.aws.greengrass.lifecyclemanager.KernelLifecycle.lambda$executeProvisioningPlugin$2(KernelLifecycle.java:224)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.nio.file.FileAlreadyExistsException: /opt/dk/greengrass/v2
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:94)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
	at java.base/java.nio.file.Files.createDirectory(Files.java:690)
	at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
	at java.base/java.nio.file.Files.createDirectories(Files.java:743)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.createFile(FleetProvisioningByClaimPlugin.java:381)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.writeFile(FleetProvisioningByClaimPlugin.java:374)
	at com.aws.greengrass.FleetProvisioningByClaimPlugin.writeCertificate(FleetProvisioningByClaimPlugin.java:365)
	... 9 more

[mpichette]

For more context, the path that the exception is referring to is a symlink to another directory. I know the nio.Path.createDirectories is not supposed to throw an exception if the directory already exists but does it throw an exception if it encounters a symlink?

[alter-mage]

Hi @mpichette, thanks for the detailed report.

The FileAlreadyExistsException is coming from Files.createDirectories() in the fleet provisioning plugin (not the nucleus itself) when it tries to create the parent directory for your certificatePath. Since /opt/dk/greengrass/v2 is a symlink, createDirectories can fail when encountering a symlink as an intermediate path component — this is a known JDK behavior (JDK-8130464).

To answer your follow-up question: yes, Files.createDirectories can throw FileAlreadyExistsException when it encounters a symlink, even though the target is a valid directory.

We attempted to reproduce this with Amazon Corretto 11.0.30 and Eclipse Temurin 11.0.30 on Amazon Linux 2023 using the same symlink layout as your setup, but were unable to trigger the failure. Given that your JDK (OpenJDK 11.0.30, January 2026) is also recent, there may be something specific to your Yocto filesystem or partition configuration contributing to this. Could you share:

• Which version of the fleet provisioning plugin you're using (custom certificatePath requires v1.2.2+)
• The output of stat -L /opt/dk/greengrass/v2 and stat /opt/dk/greengrass/v2
• The filesystem types for both /opt and /data (output of mount | grep -E '/opt|/data')
• Whether there are nested symlinks in the path

In the meantime, you can work around this by using the real path in your config — set certificatePath and privateKeyPath to /data/opt/dk/greengrass/v2/keys/... instead of the symlinked /opt/dk/greengrass/v2/keys/....