Validate TemplateProcessing on deserialization (fix panic on crafted tokenizer.json)

[v-code01]

What

TemplateProcessing deserialized from a tokenizer.json skips the builder's validation. serde routes through From<TemplateProcessingDeserializer>, which builds the struct directly without verifying that every special token referenced by the single/pair templates actually appears in special_tokens.

That invariant is not optional — apply_template indexes the special-tokens map directly:

// processors/template.rs
let tok = &self.special_tokens.0[id]; // We already checked existence above

The "already checked" only holds on the programmatic path (TemplateProcessingBuilder::build → validate). On the deserialization path it does not, so a crafted config whose template references a token missing from special_tokens loads fine and then panics at encode time on the missing map key.

Repro

A tokenizer.json post-processor like:

{
  "type": "TemplateProcessing",
  "single": [{ "SpecialToken": { "id": "[CLS]", "type_id": 0 } }],
  "pair": [],
  "special_tokens": {}
}

deserializes without error, then panics inside apply_template on the first encode.

Fix

Switch the serde bridge from from to try_from and route through TemplateProcessingBuilder, so the same validate that guards the programmatic path also guards deserialization. Invalid configs now return a clean Err at load time instead of panicking later.

#[serde(tag = "type", try_from = "TemplateProcessingDeserializer")]

impl TryFrom<TemplateProcessingDeserializer> for TemplateProcessing {
    type Error = String;
    fn try_from(t: TemplateProcessingDeserializer) -> Result<Self, Self::Error> {
        TemplateProcessingBuilder::default()
            .single(t.single)
            .pair(t.pair)
            .special_tokens(t.special_tokens)
            .build()
            .map_err(|e| e.to_string())
    }
}

Tests

Added deserialize_missing_special_tokens_is_rejected: a template referencing [CLS] with empty special_tokens now returns Err from serde_json::from_str instead of panicking on encode. Existing template_processing_serde (valid round-trip) and missing_special_tokens still pass. Full tokenizers lib suite green (203 passed); cargo fmt/clippy clean on the changed file.

cc @ArthurZucker @McPatate