Offline Timeout happening?

[wills721]

Describe the bug
I am confused about an error. I have GG configured for OffLine. My Nuc config is below. My assumption is that I should NEVER see timeouts if GG is configured for OffLine? However, I am seeing Timeout exceptions. I am not using the Moquette broker as per documentation I do not seem to need this if I simply configure Nuc as shown below. This seems contradictory? Is this a bug or is this mis configured?

Nuc Config:

 "components": {
   "aws.greengrass.Cli": {
      "componentVersion": "2.16.0"
    },
    "aws.greengrass.DiskSpooler": {
      "componentVersion": "1.0.7"
    },
      "aws.greengrass.Nucleus": {
      "componentVersion": "2.16.1",
      "configurationUpdate": {
        "merge": "{ \"logging\": {\"level\": \"WARN\", \"fileSizeLimit\": \"1MB\"}, \"deploymentPollingFrequencySeconds\": \"60\", \"fleetStatus\": {\"periodicStatusPublishIntervalSeconds\":0},  \"telemetry\": {\"enabled\": false }, \"mqtt\": { \"maxInFlightPublishes\": 5, \"keepAlive\": 300, \"maxQueuedMessages\": 100, \"keepAliveTimeoutMs\": 300000, \"pingTimeoutMs\": 60000, \"operationTimeoutMs\": 60000, \"port\": 8883, \"maxPublishRetry\": 5, \"spooler\": { \"storageType\": \"Disk\", \"pluginName\": \"aws.greengrass.DiskSpooler\", \"maxSizeInBytes\": 2621440 } } }" } },
     "aws.greengrass.TokenExchangeService": {
      "componentVersion": "2.0.3",     
      "configurationUpdate": {
        "merge": "{ \"port\": 2020, \"credentialRetryInSec\": {\"clientError\": 15, \"serverError\": 30, \"unknownError\": 45} }"
      }

    },
    "com.thio.coredevice.thio-poller-common": {
      "componentVersion": "${CUSTOM_COMPONENT_VERSION}"
    },
    "com.thio.coredevice.thio-manage-common": {
      "componentVersion": "${MANAGE_COMPONENT_VERSION}"
    },
    "com.thio.coredevice.thio-printer-common": {
      "componentVersion": "${PRINTER_COMPONENT_VERSION}"
    }
  },
  "deploymentPolicies": {
    "componentUpdatePolicy": {
      "action": "NOTIFY_COMPONENTS",
      "timeoutInSeconds": 30
    },
    "configurationValidationPolicy": {
      "timeoutInSeconds": 60
    },
    "failureHandlingPolicy": "ROLLBACK"
  }
}

Exception:
> File "/var/snap/thio/x1/greengrass/v2/packages/artifacts-unarchived/com.thio.coredevice.thio-poller-common/1.0.199/thio_bp/thioglobals.py", line 1079, in global_publish_binary_message_to_topic
    future.result(timeout=retry_interval) # 2s timeout
    │      │              └ 4
    │      └ <function Future.result at 0xffffb267bf60>
    └ <Future at 0xffffac99c860 state=running>

  File "/snap/thio/x1/usr/lib/python3.12/concurrent/futures/_base.py", line 458, in result
    raise TimeoutError()
          └ <class 'TimeoutError'>

TimeoutError

To Reproduce
Test offline

Expected behavior
Timeout exceptions should NEVER happen?

Actual behavior
Timeouts are happening

Environment

• OS: Ubuntu Core 24
• JDK version:
• Nucleus version: 2.16.1

Additional context
Add any other context about the problem here.

E.g. what is the impact of the bug?

[cookpate]

If the device never comes online, then messages spooled to disk will never be published.
I don't quite understand the nature of the timeout experienced here. The publish IPC should be fire-and-forget. Do you have Nucleus logs when this timeout happens?

[aws-kevinrickard]

Your method global_publish_binary_message_to_topic is calling future.result(timeout=retry_interval) , if it doesn't complete within the timeout period it will error.

Could you show your code or what you're trying to do?

There is no code posted so it's not clear what's going on, so the following is all speculation:

It sounds like you're using a modified version of our local pub/sub IPC example: https://docs.aws.amazon.com/greengrass/v2/developerguide/ipc-publish-subscribe.html , which includes a method called publish_binary_message_to_topic. But our example code uses the synchronous version of publish_to_topic which will not time out. If you follow that it should not time out.

But, that example is for local pub/sub IPC, notably not sending messages to an MQTT broker like Moquette. If you want to send messages to Moquette, you should be using an MQTT client. Can you link to the documentation you're following?

[wills721]

Much apologies for my delay. Here is the code I am referring to:

def global_publish_binary_message_to_topic(ipcClient, topic, message):
  
    max_retries = 3
    retry_interval = 10  # seconds
    retries = 0
    
    msg_string = json.dumps(message)
    
    while retries < max_retries:
        try:                  
            request = PublishToIoTCoreRequest()
            request.topic_name = topic
            request.payload = bytes(msg_string, "utf-8")
            request.qos = QOS.AT_LEAST_ONCE
            operation = ipcClient.new_publish_to_iot_core()
            operation.activate(request)
            future = operation.get_response()
            future.result(timeout=retry_interval) 
            return True
        except Exception as e:
            retries += 1
            if shutdownEvent.is_set():
                return False
            with logger.contextualize(device_id=gateway_id): gw_logger.exception(f"Re-trying IPC client error after publish failure.")
            
    return False

I would expect above to never fail -- ever -- if off line buffer is configured properly.

@aws-kevinrickard

[alter-mage]

Thanks for sharing the code, @wills721.

To understand what's happening under the hood when the timeout occurs, could you provide the following?

1. Nucleus logs at DEBUG level — Your current config has logging set to WARN, which suppresses the log lines we need to see. Could you temporarily set the log level to DEBUG, reproduce the timeout, and share the relevant logs?

2. The size of spooler.db — When the timeouts are happening, check the file size at {greengrass-root}/work/aws.greengrass.DiskSpooler/spooler.db. This will tell us if the spool is filling up.

3. config.tlog — Located at {greengrass-root}/config/config.tlog. This shows the effective running configuration on the device and will help us verify the spooler and MQTT settings are applied as expected.