Make Microsoft.ML.OnnxRuntimeGenAI.Tokenizer a Microsoft.ML.Tokenizers.Tokenizer #970
Conversation
stephentoub
force-pushed
the
derivetokenizer
branch
from
4464046 to
0b759a7
Compare
|
@stephentoub is this unblocked now that #987 merged? |
stephentoub
force-pushed
the
derivetokenizer
branch
from
0b759a7 to
8a306e7
Compare
|
Closing and reopening the PR to trigger all the pipelines |
|
@baijumeswani looks like the builds are failing because of issues with the feed. Can you please help unblock. |
|
@stephentoub looks like you need to resolve the conflicts. |
stephentoub
force-pushed
the
derivetokenizer
branch
from
8a306e7 to
bcbd6c8
Compare
Done |
|
@baijumeswani looks like this is ready to merge. |
|
closing/re-opening to trigger |
|
@stephentoub not sure what happened. But it seems like the build started failing. Could you please take a look? |
|
@baijumeswani Looks the |
|
trying again. |
…s.Tokenizer This enables an ONNX Runtime GenAI tokenizer instance to be used anywhere a Microsoft.ML.Tokenizers tokenizer is accepted. If we'd prefer, rather than having Tokenizer be a base class for the ONNX Runtime one, we could instead expose some sort of `public Microsoft.ML.Tokenizer.Tokenizer AsTokenizer()` conversion method that returns a wrapper object (though that's a bit confusing given the names of the type are the same, just different namespaces).
stephentoub
force-pushed
the
derivetokenizer
branch
from
bcbd6c8 to
c02f69a
Compare
|
@baijumeswani looks like this one's ready to merge. |
|
@baijumeswani looks like tests are failing on the UTF8 issue again |
This enables an ONNX Runtime GenAI tokenizer instance to be used anywhere a Microsoft.ML.Tokenizers tokenizer is accepted. If we'd prefer, rather than having Tokenizer be a base class for the ONNX Runtime one, we could instead expose some sort of
public Microsoft.ML.Tokenizer.Tokenizer AsTokenizer()conversion method that returns a wrapper object (though that's a bit confusing given the names of the type are the same, just different namespaces).cc: @luisquintanilla, @tarekgh