Avoid invalid UTF-8 characters breaking connector responses (3.x+)

[Mark-H]

Bug report

Summary

If there's some invalid UTF-8 sequences in a table that you're listing through a connector/processor, that may crap out with an error like this:

Processor failed creating output array due to JSON error 5

Now that MODX 3 requires at least 7.2, we can adjust how we encode JSON to make it more forgiving by adjusting the flags passed to json_encode in the base Processor class (outputArray() method), from:

    $output = json_encode(array(
        'success' => true,
        'total' => $count,
        'results' => $array
    ));

To:

    $output = json_encode(array(
        'success' => true,
        'total' => $count,
        'results' => $array
    ), JSON_INVALID_UTF8_SUBSTITUTE);

With that it wont fail entirely, but instead will substitute the invalid sequence with the boxed question mark.

(This can't be applied to 2.x because this flag was added in PHP 7.2.)

Step to reproduce

I don't really know how to break UTF-8 sequences on purpose to be honest. But It Happens (tm).

Observed behavior

Error message and no response.

Expected behavior

No error message, response containing the boxed question mark where the invalid character was.

Related issue(s)/PR(s)

N/a. Prompted by modmore ticket 29288.

Environment

Irrelevant.

[Jako]

I don't really know how to break UTF-8 sequences on purpose to be honest. But It Happens (tm).

There exist two unicode flavours with diacritical charcters (äöü etc.): precomposed and decomposed. Both use different byte sequences for the same character. And only one is valid in JSON, if I remember right.

There are some solutions out there: https://stackoverflow.com/questions/1089966/utf8-filenames-in-php-and-different-unicode-encodings

[JoshuaLuckers]

There are two options added in 7.2 for this.

• JSON_INVALID_UTF8_IGNORE: Ignore invalid UTF-8 characters.
• JSON_INVALID_UTF8_SUBSTITUTE: Convert invalid UTF-8 characters to \0xfffd (Unicode Character 'REPLACEMENT CHARACTER')

I'm in favor of using JSON_INVALID_UTF8_SUBSTITUTE to substitute the character instead of ignore it. Since ignoring it will remove the character sequence and might open the door for potential security issues.

[JoshuaLuckers]

I don't really know how to break UTF-8 sequences on purpose to be honest. But It Happens (tm).

User input might also cause this issue.

[JoshuaLuckers]

json_encode is used a lot throughout the code base, we need to make a list of places where it makes sense to add this option by default. E.g json_encode is also used in manager controllers like here:

revolution/manager/controllers/default/resource/update.class.php

Line 70 in 5f49a4c

Ext.onReady(function() {MODx.load(' . json_encode($data) . ')});</script>');