Deployment failure

[wills721]

Describe the bug A clear and concise description of what the bug is.

Deployment failing and I see below errors. How do I dig into this? Any clue?

Nov 24 20:12:54 thio-lab-0 thio.iotcored[58953]: D[iotcored] modules/iotcored/src/mqtt.c:644: Received suback id 36.
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: E[ggl-http] modules/ggl-http/src/gghttp_util.c:145: Curl request failed due to error: Problem with the local SSL certificate
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: I[ggl-http] modules/ggl-http/src/gghttp.c:224: HTTP code: 0
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: E[ggdeploymentd] modules/ggdeploymentd/src/deployment_handler.c:945: The listThingGroupsForCoreDevice call failed with response .
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: E[ggdeploymentd] modules/ggdeploymentd/src/deployment_handler.c:1536: Cloud call to list thing groups failed. Cloud deployment requires an updated thing group list.
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: E[ggdeploymentd] modules/ggdeploymentd/src/deployment_handler.c:2461: Failed to do dependency resolution for deployment, failing deployment.
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: T[core-bus] modules/core-bus/src/client_common.c:58: Connecting to gg_fleet_status.
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: T[ggl-sdk] /root/parts/aws-greengrass-lite/build/_deps/ggl_sdk-src/src/vector.c:103: Appended to 0xffff8f77e0f8.
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: T[core-bus] modules/core-bus/src/client_common.c:83: Writing data to gg_fleet_status.
Nov 24 20:12:54 thio-lab-0 thio.ggdeploymentd[60539]: T[core-bus] modules/core-bus/src/client.c:52: Waiting for r

To Reproduce Steps to reproduce the behavior. If possible, provide a minimal
amount of code that causes the bug.

Expected behavior A clear and concise description of what you expected to
happen.

Actual behavior Tell us what actually happened.

Environment

• OS: [e.g. Ubuntu 20.04]
• Nucleus version:
• Commit id:

Additional context Add any other context about the problem here.

E.g. what is the impact of the bug?

[rawalexe]

Same as previous it does not have sufficient cloud permissions: https://github.com/aws-greengrass/aws-greengrass-lite/blob/main/docs/Provisioning.md#create-the-iot-core-policy. If you have any AI agents then try adding https://github.com/aws-greengrass/greengrass-agent-context-pack to your agent's context it can provide you with more concise error details

[wills721]

I see. In my case I already had manually provisioned things before I started using Lite. Just tried re-creating it. But, I have always had a policy attached to my things that are now pretty open (iot:* greengrass:* and even s3:*). I don't have AI agents.

[rawalexe]

you are still facing the issue with this then? If yes can you paste your config.db contents. You can use this script: https://github.com/aws-greengrass/aws-greengrass-lite/blob/main/misc/db_to_yaml.sh.

S3 is a IAM policy that can only be changed from IAM. and others are IOT core thing policy

[wills721]

So I see this in the docs:
https://docs.aws.amazon.com/iot/latest/apireference/API_ListThingGroupsForThing.html

but i see this call in the lite logs:
I[ggl-http] modules/ggl-http/src/gghttp.c:178: Preparing call to data endpoint provided as [removed]:8443/greengrass/v2/coreDevices/thio-lab-0/thingGroups

Is that right? @rawalexe I don't find the API used in Lite anywhere (yet)

[wills721]

This is my deployment that is failing:

{
  "deploymentId": "[removed]",
  "schemaDate": "2021-05-17",
  "deploymentName": "Will-Development",
  "configurationArn": "[removed]",
  "components": {
    "aws.greengrass.Cli": {
      "version": "2.15.0"
    },
    "aws.greengrass.TokenExchangeService": {
      "version": "2.0.3"
    },
    "aws.greengrass.DiskSpooler": {
      "version": "1.0.6"
    },
    "com.thio.coredevice.thio-printer-will": {
      "version": "1.0.436"
    },
    "com.thio.coredevice.thio-manage-will": {
      "version": "1.0.436"
    },
    "aws.greengrass.Nucleus": {
      "version": "2.15.0",
      "configurationUpdate": {
        "merge": {
          "mqtt": {
            "port": 8883,
            "maxPublishRetry": 5,
            "spooler": {
              "storageType": "Disk",
              "pluginName": "aws.greengrass.DiskSpooler",
              "maxSizeInBytes": 2621440
            }
          }
        }
      }
    },
    "aws.greengrass.clientdevices.IPDetector": {
      "version": "2.2.2"
    },
    "aws.greengrass.clientdevices.Auth": {
      "version": "2.5.4"
    },
    "com.thio.coredevice.thio-poller-will": {
      "version": "1.0.437"
    }
  },
  "failureHandlingPolicy": "ROLLBACK",
  "requiredCapabilities": [],
  "componentUpdatePolicy": {
    "timeout": 30,
    "action": "NOTIFY_COMPONENTS"
  },
  "configurationValidationPolicy": {
    "timeout": 60
  }
}

[rawalexe]

These components aren't compatible with Lite, Lite doe not accept the aws.greengrass.Nucleus based deployments, If you want to change the nucleus configuration use aws.greengrass.NucleusLite Not all the Public component are compatible with lite. Look for * or aws_nucleus_lite under the runtime field of the recipe. If the runtime field isn't there then it's not compatible with lite.

Also lite only supports thing group deployments which is done by subscribing to the IOT jobs topic.

[rawalexe]

Refer to the guide here: https://github.com/aws-greengrass/aws-greengrass-lite/blob/main/docs/RECIPE_SUPPORT_CHANGES.md and the examples here: https://github.com/aws-greengrass/aws-greengrass-lite/tree/main/docs/examples/supported_lifecyle_types

[wills721]

Would that be why there is a curl error as well:

Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: E[ggl-http] modules/ggl-http/src/gghttp_util.c:145: Curl request failed due to error: Problem with the local SSL certificate
Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: I[ggl-http] modules/ggl-http/src/gghttp.c:224: HTTP code: 0
Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: E[ggdeploymentd] modules/ggdeploymentd/src/deployment_handler.c:945: The listThingGroupsForCoreDevice call failed with response .
Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: E[ggdeploymentd] modules/ggdeploymentd/src/deployment_handler.c:1536: Cloud call to list thing groups failed. Cloud deployment requires an updated thing group list.
Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: E[ggdeploymentd] modules/ggdeploymentd/src/deployment_handler.c:2461: Failed to do dependency resolution for deployment, failing deployment.
Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: T[core-bus] modules/core-bus/src/client_common.c:58: Connecting to gg_fleet_status.
Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: T[ggl-sdk] /root/parts/aws-greengrass-lite/build/_deps/ggl_sdk-src/src/vector.c:103: Appended to 0xffffa7c2e0f8.
Nov 25 22:44:04 thio-lab-0 thio.ggdeploymentd[1250]: T[core-bus] modules/core-bus/src/client_common.c:83: Writing data to gg_fleet_status.

?

@rawalexe

[rawalexe]

This looks like your device is failing to communicate with the greengrass data plane, Does your greengrass device has a policy similar to below, Specifically the greengrass,*

[rawalexe]

Also the response code 0 is very suspicious, what's your permissions on the device certificates?

[wills721]

Regarding policies yes. Wide open. greengrass:* and iot:*

Regarding permissions on pem certs -- 400

I looked at the doc you shared. I did read this previously. All my recipe's use the runtime: *

The confusing part here to me is there are recipes and there are deployments. What I shared with you above is a deployment but you shared a recipe. My recipe and my deployments are separate json files and we use GDK and also the console to do deployments. If I remove Nucleus as shown in my deployment above there is no alternative choice using the console.