Add a description of the theme server contract

[akhmerov]

This documents the contract between the engine and the theme server, and I was motivated to add it because it helps me understand the discussion in jupyter-book/myst-enhancement-proposals#34

AI disclosure

Most of the text is implemented using OpenAI codex, which I subsequently reviewed checking for completeness and overall soundness. I did not validate against the source code because my limited typescript knowledge and the need to search through the codebase make it a challenge.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 98bad46

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[akhmerov]

Some things that aren't clear:

• How should the theme serve things that aren't pages? This includes Json assets etc.
• Should we document that the theme must be able to shut down on kill without side effects?

[akhmerov]

The CI failure seems unrelated and just general flakiness.

[agoose77]

In future we should link to or author a section that describes this separation.

[akhmerov]

What do you mean? I'm not sure what is missing.