feat(tokenization): replace RenderChat with RenderChatCompletion RPC

[sagearc]

Closes #425. Rebased on top of #461.

Changes Overview

Switches Render and RenderChat in UdsTokenizer to use the new RenderCompletion and RenderChatCompletion RPCs introduced in #461, replacing the old RenderChatTemplate flow.

On the Go side, RenderChat now builds a native RenderChatCompletionRequest proto (messages, tools, chat_template_kwargs) and returns token IDs directly instead of calling Encode on a rendered prompt string. Render calls RenderCompletion with the prompt list and returns token IDs directly too — neither returns character offsets anymore since the renderer service doesn't produce them.

Protocol Design

Tools and chat_template_kwargs are both serialized as JSON strings in the proto (tools_json, chat_template_kwargs). This avoids building a typed proto structure for fields that are already arbitrary JSON at the call site ([]interface{} in GIE's ChatCompletionsRequest), and lets Python deserialize them directly without field renaming or special-casing.

On the Python side, the gRPC servicer and renderer are updated to match: the renderer service methods now accept typed ChatCompletionRequest/CompletionRequest objects directly instead of going through a JSON round-trip. Proto-to-request conversion uses MessageToDict and json.loads for the JSON string fields.

Also excludes generated pb.go and pb2 files from golangci-lint and ruff in CI.

[github-actions]

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[Copilot]

Pull request overview

This PR updates the KV-cache manager’s UDS tokenizer client to use the newer vLLM renderer RPCs (RenderChatCompletion / RenderCompletion) instead of the legacy chat-template rendering flow, and adjusts tests/protobuf bindings accordingly.

Changes:

• Switch Go UDS tokenizer client Render to call RenderCompletion and RenderChat to call RenderChatCompletion.
• Update Go and Python tests to reflect the new RPCs and response shapes (notably: offsets are no longer asserted).
• Regenerate Go protobuf/grpc bindings to include the new RPCs and message types.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/e2e/uds_tokenizer/uds_e2e_test.go	Updates e2e assertions to ignore offsets and validate determinism under the new render RPCs.
services/uds_tokenizer/tests/test_renderer.py	Adjusts integration tests for RenderChatCompletion; one assertion was weakened.
pkg/tokenization/uds_tokenizer_test.go	Updates mock server + unit tests to cover RenderChatCompletion / RenderCompletion.
pkg/tokenization/uds_tokenizer.go	Main client change: builds OpenAI-ish JSON payloads and calls new renderer RPCs.
api/tokenizerpb/tokenizer_grpc.pb.go	Regenerated gRPC client/server stubs with new RPC methods.
api/tokenizerpb/tokenizer.pb.go	Regenerated protobuf messages for new render request/response + MM feature types.
api/indexerpb/indexer_grpc.pb.go	Regenerated header/version metadata.
api/indexerpb/indexer.pb.go	Regenerated header/version metadata.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sagearc]

@vMaroon

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 8 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gyliu513]

Not yours, but I think we need a timeout in order not to block the whole grpc server.

from concurrent.futures import TimeoutError as FuturesTimeoutError

try:
    result = asyncio.run_coroutine_threadsafe(
        self.renderer_service.render_chat(chat_request, request.model_name),
        self._loop,
    ).result(timeout=30)
except FuturesTimeoutError:
    context.abort(grpc.StatusCode.DEADLINE_EXCEEDED, "render_chat timed out")

[sagearc]

Since we're already migrating to async here, would it make more sense to add it there?

[vMaroon]

Let's follow-up separately if needed.

[vMaroon]

/lgtm
/approve