Module dependency version differences

[eero-t]

This is slightly expanded version of #574, which was closed without proper explanation.

Description

Some requirements.txt files declare specific version of the dependencies, that differ from versions used in other files in this repo.

What's worse, files include no comments on why given versions have been selected. Does newer version conflict with some other package? If yes, what other package, and which version if it (maybe that conflict is already fixed in newer one)?

=> Lack of such conflict info means that when the version eventually needs to be increased, it can break unknown other things.

At worst these differences can be bugs, and at best, they cause unnecessary differentiation, that prevents more optimal creation of the GenAIComps images (using multiple levels of base images).

PS. just pulling whatever version happens to be latest one, is IMHO not really good either. It means that image functionality, or even building them can break randomly just by doing a later rebuild. IMHO ideal case would be clearly defined (staggered) base images which have fixed dependency versions that get updated & tested regularly (e.g. for every release), and images built on top of those, needing to fetch only few extras packages.

Version differences

In current Git head, Python requirement files request different versions of several Python packages:

reqs=$(git ls-files '*/requirements*.txt')
for req in $(grep -h '^[^#]*[[<~=>]' $reqs | sed 's/ *[^-_a-zA-Z0-9][<~=>].*$//' | cut -d'[' -f1 | sort -u); do
        if [ "$(grep -Eh "^$req([[ <~=>]|$)" $reqs | sed 's/ *#.*$//' | sort -u | wc -l)" -gt 1 ]; then
                echo
                grep -Eh "^$req([[ <~=>]|$)" $reqs | sed 's/ *#.*$//' | sort -u
        fi
done

accelerate
accelerate>=0.26.0

aiohttp
aiohttp==3.10.11

docarray
docarray[full]

haystack-ai
haystack-ai==2.3.1

httpx
httpx==0.27.2

huggingface-hub
huggingface-hub<=0.24.0
huggingface-hub==0.27.1
huggingface-hub==0.30.2

huggingface_hub
huggingface_hub==0.27.1

langchain
langchain==0.2.9
langchain  --extra-index-url https://download.pytorch.org/whl/cpu

langchain_community
langchain_community==0.2.7
langchain_community  --extra-index-url https://download.pytorch.org/whl/cpu

numpy
numpy==1.23.5
numpy==1.26.4
numpy<2

openai
openai==1.57.4
openai==1.81.0

opentelemetry-api
opentelemetry-api==1.27.0

opentelemetry-exporter-otlp
opentelemetry-exporter-otlp==1.27.0

opentelemetry-sdk
opentelemetry-sdk==1.27.0

pathway
pathway[xpack-llm]

pgvector
pgvector==0.2.5

predictionguard
predictionguard==2.2.1

pydantic
pydantic==2.7.2
pydantic==2.8.2
pydantic==2.9.1
pydantic==2.9.2

redis
redis==5.2.1

torch
torch==2.5.1 --index-url https://download.pytorch.org/whl/cpu

torchvision
torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cpu

transformers
transformers>=4.50.0
transformers[sentencepiece]

unstructured
unstructured[all-docs]
unstructured[all-docs] >= 0.16

[eero-t]

The optional/extra dep list for given package (project specific list ID given inside []) are not a problem, but it's good to document also why those extras are needed.

[joshuayao]

We had a discussion for the issues from dependencies last week. The solution is as follows:

• Automate Dependency Updates with Confidence: We will stop pinning dependencies or using the latest by default if possible. Instead, an automated process will continuously test OPEA against the latest dependency versions. Once validated, these "latest stable" versions will be updated to requirements.txt files.

• Document What's Necessary: If a dependency must be pinned to a specific version for a requirements.txt, the reason must be documented clearly. This explains why the pin exists and under what conditions it can be removed.

@chensuyue is working on it.