[v7.4.0] Order-Dependent Data Loss When Mixing doc.as<JsonObject>() and doc["key"] Assignments

[Larry70455]

Description
When populating a DynamicJsonDocument (tested with buffer size 512 and 1024 on ESP32), there appears to be an order-dependent issue leading to data loss during serialization.

Specifically, if the first operation on the document is obtaining a JsonObject reference via doc.as<JsonObject>() and adding members through that reference, subsequent additions made directly using the doc["key"] = value syntax seem to cause the initial members (added via the reference) to be omitted from the final serialized JSON string. This occurs even when doc.overflowed() returns false and the buffer size is demonstrably sufficient for the final object (as proven by other addition orders working correctly).

Conversely, if the first operation is adding a member directly using doc["key"] = value, subsequent additions (either via doc[] or via a JsonObject reference obtained after the first addition) serialize correctly. Adding all members via a reference obtained first seems to cause complete serialization failure (null).

Troubleshooter's report

1. The program uses ArduinoJson 7
2. The issue happens at run time
3. The issue concerns serialization
4. Output contains null
5. JsonDocument::overflowed() returns false
6. The empty or missing value is a string
7. No string contains a NUL
8. The output is a char*
9. Increasing the buffer size does not solve the issue

Environment

• Microcontroller: ESP32-WROOM-32E
• Core/Framework: ESP32 DEV MODULE
• IDE: 2.3.5

Reproduction code
Minimal sketch demonstrating the issue:

#include <Arduino.h>
#include <ArduinoJson.h> // Using ArduinoJson 7.4.0

// Define buffer size (bug observed with 512 and 1024)
const size_t JSON_BUFFER_SIZE = 512;

// Helper function similar to addFloatSafe used in original project
void addFloatSafeRef(JsonObject obj, const char* key, float value) {
  if (isnan(value) || isinf(value)) {
    obj[key] = nullptr;
  } else {
    obj[key] = value; // Send as number
  }
}

// Test Function
void runTestCase(const char* caseName, int order) {
  Serial.println("----------------------------------------");
  Serial.print("Running Test Case: "); Serial.println(caseName);
  Serial.print("Buffer Size: "); Serial.println(JSON_BUFFER_SIZE);
  Serial.printf("Order Scenario: %d\n", order);

  DynamicJsonDocument doc(JSON_BUFFER_SIZE);
  JsonObject obj;
  float floatVal1 = 3.14f; float floatVal2 = -99.99f;
  bool boolVal = true; int intVal = 12345; const char* stringVal = "Test String";

  switch(order) {
    case 1: // Add Floats FIRST using reference from as<>()
      Serial.println(" -> Adding floats FIRST via reference (obj = doc.as<JsonObject>())");
      obj = doc.as<JsonObject>(); // Get reference first
      addFloatSafeRef(obj, "float1", floatVal1);
      addFloatSafeRef(obj, "float2", floatVal2);
      doc["bool1"] = boolVal; doc["int1"] = intVal; doc["string1"] = stringVal;
      break;
    case 2: // Add Non-Float FIRST using doc[], then add floats via reference
      Serial.println(" -> Adding bool FIRST via doc[], then floats via reference");
      doc["bool1"] = boolVal; // Add non-float first
      obj = doc.as<JsonObject>(); // Get reference
      addFloatSafeRef(obj, "float1", floatVal1);
      addFloatSafeRef(obj, "float2", floatVal2);
      doc["int1"] = intVal; doc["string1"] = stringVal;
      break;
    case 3: // Get reference FIRST, add ALL via reference
       Serial.println(" -> Get reference FIRST, add ALL via reference");
       obj = doc.as<JsonObject>(); // Get reference first
       addFloatSafeRef(obj, "float1", floatVal1);
       addFloatSafeRef(obj, "float2", floatVal2);
       obj["bool1"] = boolVal; obj["int1"] = intVal; obj["string1"] = stringVal;
       break;
     case 4: // Add ALL directly using doc[]
       Serial.println(" -> Add ALL directly using doc[]");
       if (isnan(floatVal1) || isinf(floatVal1)) doc["float1"] = nullptr; else doc["float1"] = floatVal1;
       if (isnan(floatVal2) || isinf(floatVal2)) doc["float2"] = nullptr; else doc["float2"] = floatVal2;
       doc["bool1"] = boolVal; doc["int1"] = intVal; doc["string1"] = stringVal;
       break;
  }

  if (doc.overflowed()) { Serial.println(" !!! ERROR: JSON document overflowed during population! !!!"); }
  else { Serial.println(" JSON document population OK (no overflow detected)."); }

  String outputJson; size_t jsonSize = serializeJson(doc, outputJson);
  if (jsonSize == 0 && !doc.isNull()) { Serial.println(" !!! ERROR: JSON serialization failed! (Returned 0 bytes) !!!"); }
  else { Serial.println(" Serialized JSON:"); Serial.println(outputJson); Serial.print(" Serialized Size: "); Serial.println(jsonSize); }
  Serial.print(" Final Memory Usage: "); Serial.println(doc.memoryUsage());
  Serial.println("----------------------------------------");
  delay(100);
}

void setup() {
  Serial.begin(115200); while (!Serial); delay(2000);
  Serial.println("\n--- ArduinoJson Order Test ---");
  Serial.print("ArduinoJson Version: "); Serial.println(ARDUINOJSON_VERSION);
  runTestCase("Floats First (via ref)", 1);
  runTestCase("Non-Float First (direct), then Floats (via ref)", 2);
  runTestCase("All via Reference (ref obtained first)", 3);
  runTestCase("All via Direct Access", 4);
  Serial.println("\n--- Test Complete ---");
}
void loop() { }

Remarks
The issue can be worked around by ensuring the first member added to the DynamicJsonDocument uses the direct doc["key"] = value syntax, rather than obtaining a reference via doc.as() first. Alternatively, adding all elements directly using doc[] also works.

Observations:
Case 1 (Floats added first via reference, others added directly) incorrectly omits float1 and float2.
Case 3 (All added via reference obtained first) fails completely, serializing as null.
Cases 2 and 4, where the first addition is direct (doc["key"]) or all additions are direct, work correctly.

[bblanchon]

Hi @Larry70455,

as<JsonObject>() returns a reference to an existing object but doesn't create one.
This function is appropriate when extracting values from a JSON document but not when creating one.

Instead, you must use to<JsonObject>().

- obj = doc.as<JsonObject>(); // Get reference first
+ obj = doc.to<JsonObject>(); // Create first
  addFloatSafeRef(obj, "float1", floatVal1);
  addFloatSafeRef(obj, "float2", floatVal2);

By the way, replacing NaN and Infinity with null is the default behavior (see ARDUINOJSON_ENABLE_NAN), so you don't need addFloatSafeRef().

Also, don't forget to check your compiler warnings.

Best regards,
Benoit