fix(openapi): handle recursive PEP 695 type aliases in schema generation

[declaresub]

Description

Fixes #4843

A recursive PEP 695 type alias used as a handler annotation crashed OpenAPI
schema generation with RecursionError. SchemaCreator.for_type_alias_type
unwrapped the alias's __value__ and recursed with no cycle guard, so a
self-referential alias such as
type JSON = None | bool | str | list[JSON] | dict[str, JSON] expanded forever.

Litestar already guards self-referential models via a registered $ref
(#2429 / #2869), but that guard was never extended to the type-alias path
added in #3715.

Changes

• A self-referential alias is now registered as a named component and its
  recursive occurrences resolve to a $ref, mirroring how recursive models are
  handled. Non-recursive aliases are still inlined, unchanged.
• Mutually recursive aliases terminate as well: the cycle entry point becomes
  the component, and aliases reached while expanding it are inlined.
• When an alias's value is itself a single component (e.g. type Node = Container
  where Container refers back to Node), the component defers to it via allOf.
• _get_normalized_schema_key now keys TypeAliasType, which exposes __name__
  but no __qualname__.
• Once recursive aliases generate schemas, example generation became reachable
  for them and would itself recurse without bound (the example factory guards
  recursive models but not aliases). Example generation is now skipped for any
  annotation that is, or contains, a self-referential alias; non-recursive
  aliases reused within a single annotation are unaffected.

Known limitation

Mutually recursive aliases used at several independent entry points produce
valid but non-minimal components (each inlines a copy of the other's value).
This is documented in for_type_alias_type.

Tests

Added coverage in tests/unit/test_openapi/test_schema.py for direct recursion,
mutual recursion, alias-to-recursive-model (allOf), example generation on
recursive aliases, and the recursion-detection helper (including the
non-recursive-reuse case).

---

📚 Documentation preview 📚: https://litestar-org.github.io/litestar-docs-preview/4848

[codecov]

Codecov Report

❌ Patch coverage is 96.61017% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.39%. Comparing base (36c3da1) to head (e352224).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
litestar/_openapi/schema_generation/schema.py	91.30%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4848      +/-   ##
==========================================
+ Coverage   67.27%   67.39%   +0.11%     
==========================================
  Files         293      293              
  Lines       15235    15290      +55     
  Branches     1727     1741      +14     
==========================================
+ Hits        10250    10304      +54     
- Misses       4838     4839       +1     
  Partials      147      147              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[declaresub]

Relationship to #4846

Heads up for reviewers: #4846 (by @Sujit-1509) fixes the same issue (#4843) and was opened first. This PR is an alternative approach, not an attempt to duplicate it — the two differ in schema shape and in one functional respect worth weighing before choosing.

Schema shape — free-form {} vs. $ref

For type JSON = None | bool | str | float | int | list[JSON] | dict[str, JSON]:

	#4846	this PR (#4848)
Cycle handling	returns a free-form Schema()	registers a named component, recursion resolves to $ref
list[JSON] item	{} (any)	{"$ref": "#/components/schemas/JSON"}
Output	smaller, structure not preserved	faithful recursion, reusable component
Diff size	~17 LOC	larger (3 modules)

Both are valid. #4846 is simpler; the argument that a recursive JSON is semantically "any valid JSON" is reasonable. This PR mirrors how litestar already handles recursive models (#2429 / #2869, via $ref), at the cost of more machinery. That's a genuine judgement call for the maintainers.

Functional difference — example generation

Independent of the schema shape, the example factory (create_examples_for_field) also recurses without bound on a recursive alias — it guards recursive models but not aliases. So a schema-only fix still hangs with create_examples=True. I verified this against #4846's branch:

type JSON = None | bool | str | float | int | list[JSON] | dict[str, JSON]

@get("/")
async def h() -> dict[str, JSON]: ...

# OpenAPIConfig(..., create_examples=True)
app.openapi_schema.to_schema()   # #4846: hangs;  this PR: returns

This PR adds a guard so example generation is skipped for any annotation that is, or contains, a self-referential alias (non-recursive aliases reused in one annotation are unaffected). If #4846 is the preferred direction, that guard is independently useful and I'm happy to contribute it there instead.

Also covered here: mutually recursive aliases, and type Node = Container where Container refers back to Node (the component defers via allOf).

Happy to defer to #4846 and close this if the simpler shape is preferred — flagging the example-generation case so it isn't lost either way.