[DOC] Document tokenizer concurrency and shutdown semantics

[tazarov]

Summary

Document the supported concurrency model for shared tokenizer instances.

Scope

• State clearly which tokenizer operations are safe to run concurrently
• Document the shutdown and teardown contract around Close()
• Add production guidance for long-lived shared tokenizer instances in services
• Update any examples or docs that imply broader safety than the code guarantees

Acceptance criteria

• Public docs describe steady-state concurrent use accurately
• Public docs describe how Close() behaves with in-flight or future requests
• Examples and guidance are aligned with the implemented lifecycle semantics

Part of #104.

[tazarov]

Closing as duplicate of #113