Remove convert hf tokenizer to IR script in cpp.

[xipingyan]

This pull request simplifies the TokenizerImpl::setup_tokenizer logic in tokenizer_impl.cpp by removing the automatic conversion of HuggingFace tokenizers to OpenVINO format at runtime. The related code and unnecessary includes are deleted, resulting in a cleaner and more maintainable implementation.

Tokenizer conversion logic removal:

• Removed all code responsible for checking the presence of HuggingFace tokenizer files (tokenizer.json, vocab.json, merges.txt) and for invoking a Python script to convert them to OpenVINO format at runtime. This means the tokenizer conversion must now be performed outside of this function, and only existing OpenVINO tokenizer files will be loaded.

Code cleanup:

• Removed unnecessary includes (<iostream>, <algorithm>) that were only used by the deleted conversion logic.

[Copilot]

Pull request overview

This PR removes the runtime conversion path that attempted to convert HuggingFace tokenizer artifacts into OpenVINO tokenizer/detokenizer IRs inside TokenizerImpl::setup_tokenizer, simplifying the tokenizer initialization flow and eliminating the Python std::system(...) invocation.

Changes:

• Deleted HF-tokenizer detection and the runtime conversion code path (including the python -c ... command).
• Cleaned up headers previously used by the conversion logic.

Comments suppressed due to low confidence (1)

src/cpp/src/tokenizer/tokenizer_impl.cpp:352

• After removing the runtime HF->OpenVINO conversion, when neither openvino_tokenizer.xml nor openvino_detokenizer.xml exists this function proceeds and calls setup_tokenizer({nullptr, nullptr}, ...), which then returns early (intended for GGUF) and leaves the TokenizerImpl constructed but unusable until a later encode/decode assert. Consider adding an OPENVINO_ASSERT/OPENVINO_THROW here with a clear message (e.g., requiring pre-converted OpenVINO tokenizer IRs) when both files are missing for non-GGUF models.

    if (std::filesystem::exists(models_path / "openvino_tokenizer.xml")) {
        ov_tokenizer = core.read_model(models_path / "openvino_tokenizer.xml", {}, filtered_properties);
    }

    if (std::filesystem::exists(models_path / "openvino_detokenizer.xml")) {
        ov_detokenizer = core.read_model(models_path / "openvino_detokenizer.xml", {}, filtered_properties);
    }

    read_config(models_path);
    read_special_tokens_map(models_path);
    // Try to read tokenizer_config if some token ids or token str are not defined.
    read_tokenizer_config_if_necessary(models_path);
    parse_chat_template_from_file(models_path / "tokenizer_config.json", m_chat_template);
    parse_chat_template_from_file(models_path / "processor_config.json", m_chat_template);
    parse_chat_template_from_file(models_path / "chat_template.json", m_chat_template);
    parse_chat_template_from_file(models_path / "chat_template.jinja", m_chat_template);
    m_original_chat_template = m_chat_template;
    setup_tokenizer(std::make_pair(ov_tokenizer, ov_detokenizer), filtered_properties);
}

---

You can also share your feedback on Copilot code review. Take the survey.