Description
We recently experienced an issue where our required python packages had conflicts in our Ray inference jobs.
@chainlink mentioned it would be a good idea to explicitly lock all the transitive dependencies in the requirements to prevent another occurrence of this situation).
We could use something like pip-tools to automate the creation of the requirements.txt file (need to figure out the best way to handle the CPU/GPU versions).
e.g. requirements.in:
--extra-index-url https://download.pytorch.org/whl/cpu
accelerate==1.5.2
datasets==2.19.1
langcodes==3.5.0
litellm==1.63.12
loguru==0.7.3
pandas==2.2.3
pydantic==2.10.6
python-box==7.3.2
s3fs==2024.2.0
sentencepiece==0.2.0
torch==2.6.0
transformers==4.49.0
uv pip install pip-tools
uv run pip-compile requirements.in
Generates requirements.txt:
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
# pip-compile requirements.in
#
--extra-index-url https://download.pytorch.org/whl/cpu
accelerate==1.5.2
# via -r requirements.in
aiobotocore==2.21.1
# via s3fs
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiobotocore
# datasets
# fsspec
# litellm
# s3fs
aioitertools==0.12.0
# via aiobotocore
aiosignal==1.3.2
# via aiohttp
annotated-types==0.7.0
# via pydantic
anyio==4.9.0
# via
# httpx
# openai
attrs==25.3.0
# via
# aiohttp
# jsonschema
# referencing
botocore==1.37.1
# via aiobotocore
certifi==2025.1.31
# via
# httpcore
# httpx
# requests
cffi==1.17.1
# via sounddevice
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via litellm
datasets==2.19.1
# via -r requirements.in
dill==0.3.8
# via
# datasets
# multiprocess
distro==1.9.0
# via openai
filelock==3.18.0
# via
# datasets
# huggingface-hub
# torch
# transformers
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
fsspec[http]==2024.2.0
# via
# datasets
# huggingface-hub
# s3fs
# torch
h11==0.14.0
# via httpcore
httpcore==1.0.7
# via httpx
httpx==0.28.1
# via
# litellm
# openai
huggingface-hub==0.29.3
# via
# accelerate
# datasets
# tokenizers
# transformers
idna==3.10
# via
# anyio
# httpx
# requests
# yarl
importlib-metadata==8.6.1
# via litellm
jinja2==3.1.6
# via
# litellm
# torch
jiter==0.9.0
# via openai
jmespath==1.0.1
# via
# aiobotocore
# botocore
jsonschema==4.23.0
# via litellm
jsonschema-specifications==2024.10.1
# via jsonschema
langcodes==3.5.0
# via -r requirements.in
language-data==1.3.0
# via langcodes
litellm==1.63.12
# via -r requirements.in
loguru==0.7.3
# via -r requirements.in
marisa-trie==1.2.1
# via language-data
markupsafe==3.0.2
# via jinja2
mpmath==1.3.0
# via sympy
multidict==6.2.0
# via
# aiobotocore
# aiohttp
# yarl
multiprocess==0.70.16
# via datasets
networkx==3.4.2
# via torch
numpy==2.2.4
# via
# accelerate
# datasets
# openai
# pandas
# transformers
openai==1.68.0
# via litellm
packaging==24.2
# via
# accelerate
# datasets
# huggingface-hub
# transformers
pandas==2.2.3
# via
# -r requirements.in
# datasets
propcache==0.3.0
# via
# aiohttp
# yarl
psutil==7.0.0
# via accelerate
pyarrow==19.0.1
# via datasets
pyarrow-hotfix==0.6
# via datasets
pycparser==2.22
# via cffi
pydantic==2.10.6
# via
# -r requirements.in
# litellm
# openai
pydantic-core==2.27.2
# via pydantic
python-box==7.3.2
# via -r requirements.in
python-dateutil==2.9.0.post0
# via
# aiobotocore
# botocore
# pandas
python-dotenv==1.0.1
# via litellm
pytz==2025.1
# via pandas
pyyaml==6.0.2
# via
# accelerate
# datasets
# huggingface-hub
# transformers
referencing==0.36.2
# via
# jsonschema
# jsonschema-specifications
regex==2024.11.6
# via
# tiktoken
# transformers
requests==2.32.3
# via
# datasets
# huggingface-hub
# tiktoken
# transformers
rpds-py==0.23.1
# via
# jsonschema
# referencing
s3fs==2024.2.0
# via -r requirements.in
safetensors==0.5.3
# via
# accelerate
# transformers
sentencepiece==0.2.0
# via -r requirements.in
six==1.17.0
# via python-dateutil
sniffio==1.3.1
# via
# anyio
# openai
sounddevice==0.5.1
# via openai
sympy==1.13.1
# via torch
tiktoken==0.9.0
# via litellm
tokenizers==0.21.1
# via
# litellm
# transformers
torch==2.6.0
# via
# -r requirements.in
# accelerate
tqdm==4.67.1
# via
# datasets
# huggingface-hub
# openai
# transformers
transformers==4.49.0
# via -r requirements.in
typing-extensions==4.12.2
# via
# anyio
# huggingface-hub
# openai
# pydantic
# pydantic-core
# referencing
# torch
tzdata==2025.1
# via pandas
urllib3==2.3.0
# via
# botocore
# requests
wrapt==1.17.2
# via aiobotocore
xxhash==3.5.0
# via datasets
yarl==1.18.3
# via aiohttp
zipp==3.21.0
# via importlib-metadata
Just a suggestion/idea, feel free to implement however seems best practice.
Reproduction
NA
Relevant log output
Expected behavior
No dependencies or transitive dependencies are used that are not in the requirements.
System Info
NA
Have you searched for similar issues before submitting this one?
Description
We recently experienced an issue where our required python packages had conflicts in our Ray inference jobs.
@chainlink mentioned it would be a good idea to explicitly lock all the transitive dependencies in the requirements to prevent another occurrence of this situation).
We could use something like
pip-toolsto automate the creation of therequirements.txtfile (need to figure out the best way to handle the CPU/GPU versions).e.g.
requirements.in:Generates
requirements.txt:Just a suggestion/idea, feel free to implement however seems best practice.
Reproduction
NA
Relevant log output
Expected behavior
No dependencies or transitive dependencies are used that are not in the requirements.
System Info
NA
Have you searched for similar issues before submitting this one?