Skip to content

Commit f4a5608

Browse files
AlienKevinclaude
andauthored
Fix outlines task1706: pin transformers and tokenizers versions (#39)
The Dockerfile and runner.sh install transformers and tokenizers without version pins. transformers>=5.0.0 combined with tokenizers>=0.22 breaks the test suite in two ways: 1. tokenizers 0.22 turns the BPE.__init__ DeprecationWarning into a hard error, causing GPT2Tokenizer initialization to fail. 2. transformers 5.x has a code path in _from_pretrained that calls import_protobuf_decode_error() inside an except clause, which raises ImportError when protobuf is not installed — even for non-protobuf tokenizers like GPT2. Pin transformers<5 and tokenizers<0.21 in both the Dockerfile and runner.sh to restore compatibility. Verified locally: both feature 4 and feature 6 tests pass (9/9 and 11/11) with the pinned versions. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 62e6a97 commit f4a5608

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

dataset/dottxt_ai_outlines_task/task1706/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ ENV LLAMA_CPP_FORCE_CPU=1
2626
ENV PYTORCH_ENABLE_MPS_FALLBACK=1
2727
RUN uv pip install --system -e . && \
2828
uv pip install --system pytest pytest-xdist pytest_mock pytest-asyncio pytest-benchmark pytest-cov && \
29-
uv pip install --system torch transformers sentencepiece xgrammar llama-cpp-python==0.3.16 psutil
29+
uv pip install --system torch "transformers<5" "tokenizers<0.21" sentencepiece xgrammar llama-cpp-python==0.3.16 psutil
3030

3131
# Copy the runner script
3232
COPY runner.sh /usr/local/bin/

dataset/dottxt_ai_outlines_task/task1706/runner.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ echo "Installing dependencies..."
6767
uv sync 2>/dev/null || true
6868
uv pip install --system -e .
6969
uv pip install --system pytest pytest-xdist pytest_mock pytest-asyncio pytest-benchmark pytest-cov
70-
uv pip install --system torch transformers sentencepiece xgrammar llama-cpp-python==0.3.16 psutil
70+
uv pip install --system torch "transformers<5" "tokenizers<0.21" sentencepiece xgrammar llama-cpp-python==0.3.16 psutil
7171

7272
# Run tests with timeout
7373
echo "Running tests..."

0 commit comments

Comments
 (0)