[TST] Add regression coverage for concurrent tokenizer use and shutdown

[tazarov]

Summary

Add regression coverage for concurrent tokenizer usage and shutdown behavior.

Scope

• Add tests for concurrent Encode and Decode on a shared tokenizer instance
• Add regression coverage for shutdown while work is in flight
• Run the relevant coverage with the race detector
• Update CI if needed so the lifecycle regression path is exercised automatically

Acceptance criteria

• The repo contains a deterministic regression test for the lifecycle race uncovered during concurrency evaluation
• go test -race ./... (or a focused equivalent in CI) covers the affected tokenizer lifecycle paths
• The concurrency-safe contract remains protected against future regressions

Part of #104.

[tazarov]

Closing as duplicate of #112