RTC: Fuzz testing

[dmsnell]

Fuzzing the Realtime-Collaboration (RTC) feature in the editor

This tracking ticket is meant to provide an overview of detecting and proposing fixes for concurrency bugs in the RTC subsystem. This work should be part of the WordPress 7.0 release (including backports into Core), consisting of bugs/defects in the proposed beta/release-candidate code.

As this work is primarily driven by LLMs, many of the reports may be suspect until demonstrated to be real.

Findings

• RTC: potential data corruption on block merge write path due to rich-text text offset directly into a diff API that operates on HTML string indices #77532
  • RTC: fix #77532, rich text offset mismatch with diff API #77658
• RTC: add regression tests for data corruption bug due to cursor scope issue #77662
• RTC: Fix title divergence between users on page refresh after title update #77666
• RTC: fix connection lost error on large update cause by mismatch between update size bounds check and expanded base64 update size #77669
• RTC: fix cursor awareness / presence bug in nested rich text elements #77673
• RTC: Fix race condition on room creation which can cause a split update log #77675
• RTC: clients can send malformed input that causes other clients' editors to crash #77678
• RTC: Fix issue where undo acts on the wrong synced entity #77681

[danluu]

Noting a couple issues in the AI generated fixes. In #77681, the original test and fix had significant gaps. They did fix an issue, but in an incomplete way. I very strongly suspect further fuzzing would have found that issue and closed the gaps, but the initial PR wasn't very good. I would rate this below the quality I'd expect from a reasonable human, but the process here would have eventually produced a good result, just with more churn than a reasonable human would've put into the system.

#77676 was a duplicate. I'm not (attempting to) do AI generated GitHub interactions and did do any kind of AI search for dups (I used normal GitHub search and searched for different keywords), so that one is directly on me.

In terms of workflow, at one point, I tried using Conductor (conductor.build), which has been recommended by multiple people I know at other companies. This had the following issues:

• It generated PRs even though it wasn't supposed to (there's an AI process for checking that issues are valid before surfacing them to me as valid issues and it bypassed that process). These PRs were also quite broken. For example, see the busted description in [Accidental PR creation, please ignore] #77674
• Issues in different Conductor Workspaces would contaminate each other for some reason. A request to fix the CI failure in one issue would cause pushes to a different PR. Things like this happened multiple times with only a small amount of use. I confirmed that this was not a problem where I typed into the wrong window by looking at the history. Conductor Workspace isolation did not work as intended here, and issue I never had with my previous workflow
• Conductor doesn't seem to be intended to be used with what one might call "higher order" workspaces. It's easy to create workspaces "by hand" and work in each one and this can be automated in a janky way, but what I really want is to be able to have me or an agent pass it a list of issues, each one of which has some actions done to it (basically, like a map or a loop), but it's really not made for that

On the bright side, we have no known false positives (yet) that weren't automatically rejected by AI review after 4 issues were reviewed by a human who's familiar with Gutenberg (I'm not familiar with Gutenberg myself and could easily think a false positive were real if I was the only one looking at things). I'm sure our false positive rate isn't 0, so we'll get some at some point, but this fuzzer seems to be working ok at the moment and seems like a net positive even if we get a moderate number of false positives going forward.