huggingface transformers compat

[edwardcapriolo]

 @require_tokenizers
    def test_decoding_single_token(self):
        for tokenizer_class in [BertTokenizer, BertTokenizer]:
            with self.subTest(f"{tokenizer_class}"):
                tokenizer = tokenizer_class.from_pretrained("google-bert/bert-base-cased")

                token_id = 2300
                decoded_flat = tokenizer.decode(token_id)
                decoded_list = tokenizer.decode([token_id])

                self.assertEqual(decoded_flat, "Force")
                self.assertEqual(decoded_list, "Force")

                token_id = 0
                decoded_flat = tokenizer.decode(token_id)
                decoded_list = tokenizer.decode([token_id])

                self.assertEqual(decoded_flat, "[PAD]")
                self.assertEqual(decoded_list, "[PAD]")

                last_item_id = tokenizer.vocab_size - 1
                decoded_flat = tokenizer.decode(last_item_id)
                decoded_list = tokenizer.decode([last_item_id])

                self.assertEqual(decoded_flat, "##：")
                self.assertEqual(decoded_list, "##：")

The problem(s) we can run into

         ModelFetcher fetch = new ModelFetcher("google-bert", "bert-base-cased");
            File f = fetch.maybeDownload();
        BertTokenizer bt = new BertTokenizer(f.toPath());
        //https://github.com/huggingface/transformers/blob/83fe012d58528a14ee0bb0146885f2d6fcb1ec3f/tests/tokenization/test_tokenization_utils.py#L129
        //from pretrained does not include leading space here
        Assertions.assertEquals(" Force", bt.decode(2300));
        Assertions.assertEquals(" [PAD]", bt.decode(0));
        int last = bt.getModel().vocabLookup.size() - 1;
        System.out.println(bt.getModel().vocabLookup);
        Assertions.assertEquals("##:", bt.decode(last));

Our tokenizer DOES NOT strip the leading spaces but DOES " remove ##"

I have started working on a complete "PORT" of the tokenizers. As we aren't dealing with permuations of kwargs. Outside of the "it works" use cases we have it is very hard to cross verify anything. It seems like we need this, long running issues like the chinese characters and QWEN. To me to not we cant solve with "with hacks" small fixes here and there will become more complicated as we bring more models. The more we can do to make the interfaces like for like the more velocity we will get.