Feature Request: Filter Key by Size?

[dmadison]

Hello again!

The filtering feature is incredibly useful, although at the moment it's currently limited to including / excluding keys. Would it be possible to extend the filtering feature to limit values to a given size?

My suggested API would be the same as the current filtering API, except that instead of setting a key to 'true' to save it you would set the key to the number of characters to parse. Using JsonFilterExample.ino as a demo:

  StaticJsonDocument<200> filter;
  filter["list"][0]["dt"] = 10;  // max of 10 characters long (excluding terminator)
  filter["list"][0]["main"]["temp"] = 4;  // ibid, 4 characters long

Key values that are larger than the specified amount would be truncated.

The benefit would be that it would be easier to control the size of the JSON document, as unexpectedly large fields would no longer take up additional memory.

[bblanchon]

Hi @dmadison,

There have been several contradicting requests to extend the filtering feature, but they all focused on arrays (#1316, #1478, #1486, #1598). So that's the first time someone asks to restrict the length of strings.

I understand your goal, but I have the following objections:

1. it would only work for string values and not for keys
2. the syntax is obscure and inconsistent: currently, we use an object to filter objects and an array to filter array, but with this feature, we would need to use an integer to filter a string
3. too few users (probably only one) would use this feature.

I'm willing to augment the filtering feature, but we need to find something more versatile.

Best regards,
Benoit

[dmadison]

Thanks for the quick response, Benoit. A few thoughts on your objections:

1. it would only work for string values and not for keys

I'm not sure I understand your objection here. Aren't the JSON keys already filtered by the presence or lack of keys in the filtering document? Perhaps I'm misusing or misunderstanding the word "key" here.

2. the syntax is obscure and inconsistent: currently, we use an object to filter objects and an array to filter array, but with this feature, we would need to use an integer to filter a string

This is a good point. It makes sense to me but I can see it being confusing, especially in the context of array filtering.

The other alternatives I can think of (e.g. key = "strlen(n)") are more memory intensive for the filtering document and would require more processing during deserialization. I wonder if another type of JsonVariant may be a better solution, although I admit I'm not well versed in the inner workings of the library.

In place of filtering the other idea I had was to create an additional DeserializationOption for a callback function that takes a JsonVariantConst reference and is invoked whenever an object is parsed, in which case the document (in theory) only needs to be as large as the largest object. Although admittedly this creates its own memory issues if the data is not copied in the callback.

3. too few users (probably only one) would use this feature.

Haha fair enough 😛

I understand that this feature is not a good fit for the library at this time; feel free to close this issue if you wish.

---

For context, I'm dealing with an API that can return arbitrary user strings (e.g. user profile descriptions) which vary wildly in length. They also unfortunately sit in the middle of the JSON response so if the document is undersized the more important data won't be deserialized.

My current solution is to dramatically oversize the JSON document, although that's predictably not an efficient or robust long-term fix.

[bblanchon]

You're right about point 1; I withdraw this objection.

I thought about the custom query language like "truncate(10)". This sounds like a good idea, but it would require some thinking to find a versatile syntax, and the size of the library could increase dramatically.
Alternatively, we would use a regex, but it sounds overkill and would require embedding a regex library in ArduinoJson, which I'd like to avoid.
Moreover, I would probably have to extract the complete string before evaluating the expression, so you could still get a NoMemory error if the string is very long (ArduinoJson stores the temporary string in the JsonDocument).

I also thought about adding callable type value for JsonVariant, like a std::function<bool(JsonVariantConst)>.
It would be extremely versatile, but the problem is that the parser would have to extract the nested value (whether it's a string, an array, or an object) before calling the function, which defeats the purpose of the filter.