Address PR review feedback for CountTokensTask introduction

[Copilot]

Fixes correctness issues, silent failures, and missing test coverage flagged in the review of the CountTokensTask PR.

Bug fixes

• Infinite loop protection (HierarchicalChunkerTask): Upfront validation now throws if reservedTokens >= maxTokensPerChunk or overlapTokens >= effectiveMaxTokens; the chunking loop also clamps endOffset = Math.max(boundary, startOffset + 1) to guarantee forward progress even when a single character exceeds the token budget
• Silent CountTokensTask failures (HierarchicalChunkerTask): Wrapped provider calls in try/catch to fall back to estimateTokens instead of propagating errors from unsupported providers
• Tiktoken encoder hotspot (OpenAI_JobRunFns): Replaced per-call create/free with a module-level _encoderCache map keyed by model name; unknown models share the cached cl100k_base encoder

Schema corrections

• HierarchicalChunkerTask: Added doc_id to required (was runtime-validated but invisible to schema consumers)
• ContextBuilderTask: totalTokens description corrected to "Estimated token count"; docstring purged of references to non-existent countTokensModel/buildCountTokensFn
• HierarchicalChunkerTask: Docstring updated to reference CountTokensTask and estimateTokens accurately

Test coverage

Added focused unit tests for the three new provider run functions:

• OpenAI_CountTokens — verifies known-model encoder selection and cl100k_base fallback path
• Anthropic_CountTokens — asserts messages.countTokens call shape and input_tokens → count mapping
• Gemini_CountTokens — asserts countTokens call and totalTokens → count mapping

Test regression fix

Restored the rate-limiter assertion to >= 12 (was loosened to >= 10 in the original PR).

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[sroussey]

revert, there are no models that register that way.

[Copilot]

Reverted in 8d7687c — CountTokensTask is back to TypeModel("model").