Skip to content

fix(api): utf8mb4 end-to-end + drop xajax-era entity decode in ban/comms flows - #1119

Merged
rumblefrog merged 2 commits into
mainfrom
fix/issue-1108-utf8mb4-names
May 4, 2026
Merged

fix(api): utf8mb4 end-to-end + drop xajax-era entity decode in ban/comms flows#1119
rumblefrog merged 2 commits into
mainfrom
fix/issue-1108-utf8mb4-names

Conversation

@rumblefrog

Copy link
Copy Markdown
Member

Closes #1108.
Closes #1060. Closes #971. Closes #765.

Summary

  • Default DB_CHARSETutf8mb4 in web/init.php, web/config.php.template, web/phpstan-bootstrap.php, and the installer/updater templates so the panel's PDO handle issues SET NAMES utf8mb4 on every connection, matching the SourceMod plugin (sbpp_main.sp, sbpp_comms.sp) and the dev/CI schema charset. The 3-byte utf8 alias rejected supplementary-plane bytes and surfaced as Query_AddBlockInsert failed: Incorrect string value for the comms insert.
  • Drop the legacy htmlspecialchars_decode / html_entity_decode dance from every API handler that reads a name / reason / chat message from a JSON body (api_bans_add, api_bans_paste, api_bans_kick_player, api_bans_send_message, api_bans_view_community, api_comms_add, api_comms_paste). xajax HTML-encoded payloads in transit; the JSON dispatcher does not. Decoding again collapsed a user-typed literal &amp; into & and, combined with Smarty's auto-escape default (Enable automatic HTML escaping in Smarty templates #1087), produced the reported Can't get player info for <name> failure path.
  • Replace compareSanitizedString($a, $b) with plain $a === $b at the five call sites and delete the helper from system-functions.php. Both sides of the comparison are raw UTF-8; wrapping them in htmlspecialchars first was only ever a no-op (or — for invalid UTF-8 — a source of false positives).
  • Harden Api::dispatch() to pass JSON_INVALID_UTF8_SUBSTITUTE to json_encode. A single bad byte in a xpaw/php-source-query hostname/player name no longer collapses the whole response to false (the root cause of 1.8.0 Server webpage breaks if players have certain special characters. #971's "bad response" on the per-server tile).
  • BanFlowTest grows a data-driven testBanRoundTripPreservesUnicodeAndAngleBrackets (four fixtures: CJK, angle brackets, Cyrillic+emoji, literal &amp;), a testCommsAddPreservesUnicodeAndAngleBrackets, and a dispatcher-level testDispatcherSubstitutesInvalidUtf8InResponse guard.
  • Doc sync: AGENTS.md anti-patterns list + ARCHITECTURE.md legacy-patterns table pick up the two new rows (no-decode-on-JSON-params, utf8mb4-end-to-end); the Database section in ARCHITECTURE.md calls out the charset convention explicitly.

Acceptance criteria

  • Banning a connected player named 叮叮当当 succeeds via the panel.testBanRoundTripPreservesUnicodeAndAngleBrackets data set cjk nickname (web/tests/integration/BanFlowTest.php:124) drives api_bans_add and asserts the raw string survives into sb_bans.name.
  • Banning a player named =[BSID]= ethzero <Msg> succeeds and the <Msg> survives in the DB and renders escaped (not literal) in the ban list page. — Same dataprovider, angle brackets set. Round-trip asserts ban['name'] === "=[BSID]= ethzero <Msg>" (web/tests/integration/BanFlowTest.php:129). page_bans.tpl renders {$ban.player|smarty_stripslashes} under setEscapeHtml(true), so the stored <Msg> comes out as &lt;Msg&gt; in HTML (no nofilter was involved; see web/themes/default/page_bans.tpl:90,131).
  • The per-server page loads when one of those names is connected.api_servers_host_players passes player names verbatim into the JSON envelope; Api::dispatch now encodes with JSON_INVALID_UTF8_SUBSTITUTE (web/includes/Api.php:192), so a malformed or legacy-encoded byte sequence no longer produces an empty body. testDispatcherSubstitutesInvalidUtf8InResponse (web/tests/integration/BanFlowTest.php:177) guards the encoder behaviour.
  • Comms block insert from the plugin works for the same names. — Plugin already runs SET NAMES utf8mb4 (game/addons/sourcemod/scripting/sbpp_main.sp:1098, sbpp_comms.sp:1384). The panel side now aligns: DB_CHARSET defaults to utf8mb4 in web/init.php:110, web/config.php.template:39, and web/phpstan-bootstrap.php:32; sb_bans.name, sb_bans.reason, sb_comms.name, and sb_comms.reason pick up character set utf8mb4 from the {charset} placeholder in web/install/includes/sql/struc.sql. testCommsAddPreservesUnicodeAndAngleBrackets (web/tests/integration/BanFlowTest.php:148) asserts 叮叮当当 <foo> round-trips through api_comms_add into both gag and mute rows.
  • New integration test in web/tests/integration/BanFlowTest.php (or a sibling) asserts the round-trip and lands green in CI. — See the three new tests above (web/tests/integration/BanFlowTest.php:67–183). All 43 PHPUnit cases pass locally.

Test plan

  • ./sbpp.sh phpstan[OK] No errors (level 5, 168 files analysed, dba enabled against the live MariaDB).
  • ./sbpp.sh test — 43 tests / 124 assertions, all green. New cases: Ban round trip preserves unicode and angle brackets ×4, Comms add preserves unicode and angle brackets, Dispatcher substitutes invalid utf 8 in response.
  • ./sbpp.sh ts-check — silent pass (tsc --noEmit --checkJs on web/scripts/).
  • ./sbpp.sh composer api-contract — regenerates web/scripts/api-contract.js; git diff clean (no handler signatures, perm masks, or registry entries changed).

Made with Cursor

…mms flows

The JSON dispatcher now replaces legacy-encoded player/host name bytes
with U+FFFD (JSON_INVALID_UTF8_SUBSTITUTE) so a single bad byte from a
gameserver rcon response can no longer collapse the per-server admin
tile into an empty body (#971). The xajax-era `htmlspecialchars_decode`
/ `html_entity_decode` calls in `bans.add`, `bans.paste`,
`bans.send_message`, `bans.view_community`, `comms.add`, and
`comms.paste` are removed; the JSON API delivers raw UTF-8 and the
Smarty auto-escape layer (#1087) handles display-time escaping, so
decoding again was both a no-op for the common case and actively
harmful for literal `&amp;` input (#1060). `compareSanitizedString`
becomes straight `===` everywhere it was used.

The panel's PDO `DB_CHARSET` default flips from `utf8` (3-byte alias)
to `utf8mb4` in `init.php`, `config.php.template`, the installer, the
updater, and the PHPStan bootstrap, matching the SourceMod plugin's
`SET NAMES utf8mb4` (#1052) and the dev/CI stack's schema charset —
eliminating the `Incorrect string value` insert failures on
supplementary-plane characters (#765).

BanFlowTest gains a multi-byte + angle-bracket round-trip (four data
sets: CJK, angle brackets, Cyrillic+emoji, literal `&amp;`), a comms
round-trip, and a JSON-encode-substitute guard. AGENTS.md and
ARCHITECTURE.md pick up the new anti-patterns / legacy-pattern rows.

Closes #1108. Closes #1060. Closes #971. Closes #765.

Co-authored-by: Cursor <cursoragent@cursor.com>
@rumblefrog
rumblefrog force-pushed the fix/issue-1108-utf8mb4-names branch from 071a77c to 838d2d7 Compare May 4, 2026 04:13
- Extract Api::encodeEnvelope() so the U+FFFD substitution is
  observable in-process; dispatch() delegates to it. Replace the
  weak testDispatcherSubstitutesInvalidUtf8InResponse (which only
  re-asserted that json_encode accepts the flag combo) with a
  focused testEncodeEnvelopeSubstitutesInvalidUtf8 that decodes the
  encoder output and pins the U+FFFD byte sequence end-to-end, plus
  a sibling testHandleEmptyBodyProducesValidErrorEnvelope for the
  dispatcher's outer error path.
- Swap BanFlowTest's @dataProvider doc-comment for #[DataProvider]
  attribute (PHPUnit 11 deprecation; suite's only remaining one).
- web/init.php: comment now points at web/updater/data/600.php (the
  table-charset migrator + config.php rewriter), not upgrade.php
  (which writes SB_SECRET_KEY and has nothing to do with charsets).
- groups.edit handler: drop the xajax-era html_entity_decode from
  the JSON-decode of overrides / new_override. The JS still
  JSON.stringify's both, so json_decode stays for now.
- servers.send_rcon handler: document the intentional
  html_entity_decode as a deliberate carve-out from AGENTS.md's
  no-entity-decode-on-JSON-params rule -- the decoded value is
  only substring-matched against \`rcon_password\` and never
  stored or rendered, so the literal-typed-entity bypass attack
  still gets caught.
@rumblefrog
rumblefrog added this pull request to the merge queue May 4, 2026
Merged via the queue into main with commit 1aba8df May 4, 2026
4 checks passed
@rumblefrog
rumblefrog deleted the fix/issue-1108-utf8mb4-names branch May 4, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant