Skip to content

Commit e6d6383

Browse files
speediedanspeediedan
committed
provisionally deprecate pl_adjust_versions.py module, update docker build and prepare for PT 2.8 bump
1 parent 4b6dc5b commit e6d6383

File tree

11 files changed

+82
-82
lines changed

11 files changed

+82
-82
lines changed

‎.azure-pipelines/gpu-tests.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ jobs:
4646
strategy:
4747
matrix:
4848
'PyTorch | latest':
49-
image: "speediedan/finetuning-scheduler:py3.12-pt2.7.0-pl2.7-azpl-init"
49+
image: "speediedan/finetuning-scheduler:py3.12-pt2.7.1-pl2.7-azpl-init"
5050
scope: ""
5151
# how long to run the job before automatically cancelling
5252
timeoutInMinutes: "100"

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,9 @@ python collect_env_details.py
3939
You can also fill out the list below manually.
4040
-->
4141

42-
- Fine-Tuning Scheduler Version (e.g., 2.7.0):
43-
- Lightning Version (e.g., 2.7.0):
44-
- PyTorch Version (e.g., 2.7.0):
42+
- Fine-Tuning Scheduler Version (e.g., 2.7.1):
43+
- Lightning Version (e.g., 2.7.1):
44+
- PyTorch Version (e.g., 2.7.1):
4545
- Python version (e.g., 3.12):
4646
- OS (e.g., Linux):
4747
- CUDA/cuDNN version:

‎.github/workflows/release-docker.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ jobs:
3131
matrix:
3232
# initially building only the latest supported configuration
3333
python_version: ["3.12"]
34-
pytorch_version: ["2.7.0"]
34+
pytorch_version: ["2.7.1"]
3535
cust_base: ["cu12.8.0-"]
3636
pl_version: ["2.7"]
3737
steps:

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,7 @@ To ensure maximum stability, the latest Lightning patch release fully tested wit
173173
<details>
174174
<summary>Current build statuses for Fine-Tuning Scheduler </summary>
175175

176-
| System / (PyTorch/Python ver) | 2.3.1/3.9 | 2.7.0/3.9, 2.7.0/3.12 |
176+
| System / (PyTorch/Python ver) | 2.3.1/3.9 | 2.7.1/3.9, 2.7.1/3.12 |
177177
| :---------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
178178
| Linux \[GPUs\*\*\] | - | [![Build Status](https://dev.azure.com//speediedan/finetuning-scheduler/_apis/build/status/Multi-GPU%20&%20Example%20Tests?branchName=main)](https://dev.azure.com/speediedan/finetuning-scheduler/_build/latest?definitionId=1&branchName=main) |
179179
| Linux (Ubuntu 22.04) | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main&event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?branch=main&event=push)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) |

‎dockers/base-cuda/Dockerfile‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ ARG OS_VER=ubuntu22.04
1717
FROM nvidia/cuda:${CUDA_VERSION}-devel-${OS_VER}
1818

1919
ARG PYTHON_VERSION=3.12
20-
ARG PYTORCH_VERSION=2.7.0
20+
ARG PYTORCH_VERSION=2.7.1
2121
ARG CUST_BUILD=0
2222
ARG MKL_THREADING_LAYER=GNU
2323

@@ -79,9 +79,11 @@ RUN \
7979
. /tmp/venvs/fts_dev/bin/activate && \
8080
# set particular PyTorch version by default
8181
if [[ "${CUST_BUILD}" -eq 0 ]]; then \
82-
python ./requirements/pl_adjust_versions.py requirements/base.txt ${PYTORCH_VERSION}; \
83-
python ./requirements/pl_adjust_versions.py requirements/extra.txt ${PYTORCH_VERSION}; \
84-
python ./requirements/pl_adjust_versions.py requirements/examples.txt ${PYTORCH_VERSION}; \
82+
CUDA_VERSION_MM=${CUDA_VERSION%.*}; \
83+
pip install torch --no-cache-dir \
84+
--find-links="https://download.pytorch.org/whl/cu${CUDA_VERSION_MM//'.'/''}/torch_stable.html" \
85+
--find-links="https://download.pytorch.org/whl/test/cu${CUDA_VERSION_MM//'.'/''}/torch" \
86+
--find-links="https://download.pytorch.org/whl/test/cu${CUDA_VERSION_MM//'.'/''}/pytorch-triton"; \
8587
else \
8688
# or target a specific cuda build, by specifying a particular index url w/...
8789
# ... default channel
@@ -90,10 +92,8 @@ RUN \
9092
# pip install torch==1.11.1+cu113 torchvision==0.11.3+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html; \
9193
# ... pytorch nightly dev version
9294
#pip install --pre torch==2.7.0.dev20250201 torchvision==0.22.0.dev20250201 --index-url https://download.pytorch.org/whl/nightly/cu128; \
93-
# temporarily remove torchvision from the nightly build until it supports cu128 in nightlies
94-
#pip install --pre torch==2.7.0.dev20250201 --index-url https://download.pytorch.org/whl/nightly/cu128; \
9595
# ... test channel
96-
pip install --pre torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/test/cu128; \
96+
pip install --pre torch==2.8.0 --index-url https://download.pytorch.org/whl/test/cu128; \
9797
fi && \
9898
# We avoid installing Lightning and other dependencies here as they are usually upgraded anyway later in
9999
# CI but we may re-enable in the future.

‎dockers/docker_images_main.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ maybe_build(){
4343
build_eval(){
4444
# latest PyTorch image supported by release
4545
# see CUDA_ARCHES_FULL_VERSION for the full version of the pytorch-provided toolkit
46-
declare -A iv=(["cuda"]="12.8.0" ["python"]="3.12" ["pytorch"]="2.7.0" ["lightning"]="2.7" ["cust_build"]="1")
46+
declare -A iv=(["cuda"]="12.8.0" ["python"]="3.12" ["pytorch"]="2.7.1" ["lightning"]="2.7" ["cust_build"]="1")
4747
export latest_pt="base-cu${iv["cuda"]}-py${iv["python"]}-pt${iv["pytorch"]}-pl${iv["lightning"]}"
4848
export latest_azpl="py${iv["python"]}-pt${iv["pytorch"]}-pl${iv["lightning"]}-azpl-init"
4949
maybe_build iv "${latest_pt}" "${latest_azpl}"

‎dockers/docker_images_release.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ maybe_build(){
4141

4242
build_eval(){
4343
# latest PyTorch image supported by release
44-
declare -A iv=(["cuda"]="12.8.0" ["python"]="3.12" ["pytorch"]="2.7.0" ["lightning"]="2.7" ["cust_build"]="0")
44+
declare -A iv=(["cuda"]="12.8.0" ["python"]="3.12" ["pytorch"]="2.7.1" ["lightning"]="2.7" ["cust_build"]="0")
4545
export latest_pt="base-cu${iv["cuda"]}-py${iv["python"]}-pt${iv["pytorch"]}-pl${iv["lightning"]}"
4646
export latest_azpl="py${iv["python"]}-pt${iv["pytorch"]}-pl${iv["lightning"]}-azpl-init"
4747
maybe_build iv "${latest_pt}" "${latest_azpl}"

‎dockers/fts-az-base/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# limitations under the License.
1212

1313
ARG PYTHON_VERSION=3.12
14-
ARG PYTORCH_VERSION=2.7.0
14+
ARG PYTORCH_VERSION=2.7.1
1515
ARG LIGHTNING_VERSION=2.7
1616
ARG CUST_BASE
1717

‎dockers/release/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# limitations under the License.
1212

1313
ARG PYTHON_VERSION=3.12
14-
ARG PYTORCH_VERSION=2.7.0
14+
ARG PYTORCH_VERSION=2.7.1
1515
ARG LIGHTNING_VERSION=2.7
1616
ARG CUST_BASE
1717

‎requirements/pl_adjust_versions.py‎

Lines changed: 63 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -1,63 +1,63 @@
1-
import os
2-
import re
3-
import sys
4-
from typing import Dict, Optional
5-
6-
# IMPORTANT: this list needs to be sorted in reverse
7-
VERSIONS = [
8-
dict(torch="2.8.0", torchvision="0.23.0"), # nightly
9-
dict(torch="2.7.1", torchvision="0.22.1"), # stable
10-
dict(torch="2.7.0", torchvision="0.22.0"),
11-
dict(torch="2.6.0", torchvision="0.21.0"),
12-
dict(torch="2.5.1", torchvision="0.20.1"),
13-
dict(torch="2.5.0", torchvision="0.20.0"),
14-
dict(torch="2.4.1", torchvision="0.19.1"),
15-
dict(torch="2.4.0", torchvision="0.19.0"),
16-
]
17-
18-
19-
def find_latest(ver: str) -> Dict[str, str]:
20-
# drop all except semantic version
21-
ver = re.search(r"([\.\d]+)", ver).groups()[0] # type: ignore[union-attr]
22-
# in case there remaining dot at the end - e.g "1.9.0.dev20210504"
23-
ver = ver[:-1] if ver[-1] == "." else ver
24-
print(f"finding ecosystem versions for: {ver}")
25-
26-
# find first match
27-
for option in VERSIONS:
28-
if option["torch"].startswith(ver):
29-
return option
30-
31-
raise ValueError(f"Missing {ver} in {VERSIONS}")
32-
33-
34-
def main(req: str, torch_version: Optional[str] = None) -> str:
35-
if not torch_version:
36-
import torch
37-
38-
torch_version = torch.__version__
39-
assert torch_version, f"invalid torch: {torch_version}"
40-
41-
# remove comments and strip whitespace
42-
req = re.sub(rf"\s*#.*{os.linesep}", os.linesep, req).strip()
43-
44-
latest = find_latest(torch_version)
45-
for lib, version in latest.items():
46-
replace = f"{lib}=={version}" if version else ""
47-
req = re.sub(rf"\b{lib}(?!\w).*", replace, req)
48-
49-
return req
50-
51-
52-
if __name__ == "__main__":
53-
if len(sys.argv) == 3:
54-
requirements_path, torch_version = sys.argv[1:]
55-
else:
56-
requirements_path, torch_version = sys.argv[1], None # type: ignore[assignment]
57-
58-
with open(requirements_path) as fp:
59-
requirements = fp.read()
60-
requirements = main(requirements, torch_version)
61-
print(requirements) # on purpose - to debug
62-
with open(requirements_path, "w") as fp:
63-
fp.write(requirements)
1+
# import os
2+
# import re
3+
# import sys
4+
# from typing import Dict, Optional
5+
6+
# # IMPORTANT: this list needs to be sorted in reverse
7+
# VERSIONS = [
8+
# dict(torch="2.8.0", torchvision="0.23.0"), # nightly
9+
# dict(torch="2.7.1", torchvision="0.22.1"), # stable
10+
# dict(torch="2.7.0", torchvision="0.22.0"),
11+
# dict(torch="2.6.0", torchvision="0.21.0"),
12+
# dict(torch="2.5.1", torchvision="0.20.1"),
13+
# dict(torch="2.5.0", torchvision="0.20.0"),
14+
# dict(torch="2.4.1", torchvision="0.19.1"),
15+
# dict(torch="2.4.0", torchvision="0.19.0"),
16+
# ]
17+
18+
19+
# def find_latest(ver: str) -> Dict[str, str]:
20+
# # drop all except semantic version
21+
# ver = re.search(r"([\.\d]+)", ver).groups()[0] # type: ignore[union-attr]
22+
# # in case there remaining dot at the end - e.g "1.9.0.dev20210504"
23+
# ver = ver[:-1] if ver[-1] == "." else ver
24+
# print(f"finding ecosystem versions for: {ver}")
25+
26+
# # find first match
27+
# for option in VERSIONS:
28+
# if option["torch"].startswith(ver):
29+
# return option
30+
31+
# raise ValueError(f"Missing {ver} in {VERSIONS}")
32+
33+
34+
# def main(req: str, torch_version: Optional[str] = None) -> str:
35+
# if not torch_version:
36+
# import torch
37+
38+
# torch_version = torch.__version__
39+
# assert torch_version, f"invalid torch: {torch_version}"
40+
41+
# # remove comments and strip whitespace
42+
# req = re.sub(rf"\s*#.*{os.linesep}", os.linesep, req).strip()
43+
44+
# latest = find_latest(torch_version)
45+
# for lib, version in latest.items():
46+
# replace = f"{lib}=={version}" if version else ""
47+
# req = re.sub(rf"\b{lib}(?!\w).*", replace, req)
48+
49+
# return req
50+
51+
52+
# if __name__ == "__main__":
53+
# if len(sys.argv) == 3:
54+
# requirements_path, torch_version = sys.argv[1:]
55+
# else:
56+
# requirements_path, torch_version = sys.argv[1], None # type: ignore[assignment]
57+
58+
# with open(requirements_path) as fp:
59+
# requirements = fp.read()
60+
# requirements = main(requirements, torch_version)
61+
# print(requirements) # on purpose - to debug
62+
# with open(requirements_path, "w") as fp:
63+
# fp.write(requirements)

0 commit comments

Comments
 (0)