Add Stable Diffusion 3.5 Medium support

[zlaazlaa]

No description provided.

[zlaazlaa]

Successfully ran on the Linux platform with NVIDIA GeForce RTX 3090 Ti and Qualcomm Snapdragon 8 Elite Gen5 24G (CPU, 8-bit quantization) .

[wangzhaode]

Do not introduce SentencePiece as a third-party library. It is preferable to use MNN's internal implementation for the tokenizer.

[zlaazlaa]

Do not introduce SentencePiece as a third-party library. It is preferable to use MNN's internal implementation for the tokenizer.

Hi, SentencePiece was previously introduced due to the lack of Unigram model support in the internal MNN tokenizer.
Now I add Unigram support internally, eliminating the need for the external dependency.

[wangzhaode]

I'm working on an optimized refactor of the MNN tokenizer. I recommend replacing the current approach with MNN's internal implementation once it's available. Would it be possible to wait for the tokenizer update to be released before merging this PR?

[zlaazlaa]

I'm working on an optimized refactor of the MNN tokenizer. I recommend replacing the current approach with MNN's internal implementation once it's available. Would it be possible to wait for the tokenizer update to be released before merging this PR?

That's great. Will the optimized tokenizer merge the current two tokenizers?

./transformers/llm/engine/src/tokenizer.cpp
./transformers/diffusion/engine/src/tokenizer.cpp

[wangzhaode]

I'm working on an optimized refactor of the MNN tokenizer. I recommend replacing the current approach with MNN's internal implementation once it's available. Would it be possible to wait for the tokenizer update to be released before merging this PR?

That's great. Will the optimized tokenizer merge the current two tokenizers?

./transformers/llm/engine/src/tokenizer.cpp
./transformers/diffusion/engine/src/tokenizer.cpp

Yes.