Skip to content

Commit 638212f

Browse files
lithorusrobin-ubisoft-helix
lithorus
and
robin-ubisoft-helix
authored
Split proto into it's own package and create packages for all python modules (#1681)
Summary of changes : ### Proto split - Split proto files into it's own folder (proto) and package (opencue_proto) ### Packaging - Removed `build_proto.sh` script - Replace `setup.py` and `requirements.txt` with `pyproject.toml` with proper dependencies for testing, building and installation - Prefixed all packages with "opencue_" to not clash with pypi packages. - Uses versioningit for versiosing the packages. If current commit is tagged, it will use that as version name. Otherwise it will add how many commits since last tag and current commit hash. Eg. `1.4.11.post156+g23df172.d20250416` ### CI/CD - Change test workflow from using PYTHONPATH to use installed packages of dependencies. - Use pre-built packages (built with python 3.7) for use in CI/CI testing. (with workflow dependencies) - Added test for installing all packages using python 3.11 and python 3.12 (can easily add more) - Test packages using `python -m pytest ${package}` which uses definded test dirs in `pyproject.toml` --------- Signed-off-by: Jimmy Christensen <[email protected]> Co-authored-by: Robin (Ubisoft) <[email protected]>
1 parent ae008d1 commit 638212f

File tree

135 files changed

+1596
-1419
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

135 files changed

+1596
-1419
lines changed

‎.github/workflows/testing-pipeline.yml‎

Lines changed: 195 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,145 @@ on:
77
branches: [ master ]
88

99
jobs:
10+
build_opencue_packages:
11+
name: Build opencue packages
12+
runs-on: ubuntu-22.04
13+
container: python:3.7
14+
outputs:
15+
opencue_proto_path: ${{ steps.build_opencue_proto.outputs.opencue_proto_path }}
16+
opencue_pycue_path: ${{ steps.build_pycue.outputs.opencue_pycue_path }}
17+
opencue_pyoutline_path: ${{ steps.build_pyoutline.outputs.opencue_pyoutline_path }}
18+
opencue_cueadmin_path: ${{ steps.build_cueadmin.outputs.opencue_cueadmin_path }}
19+
opencue_cuesubmit_path: ${{ steps.build_cuesubmit.outputs.opencue_cuesubmit_path }}
20+
opencue_rqd_path: ${{ steps.build_rqd.outputs.opencue_rqd_path }}
21+
steps:
22+
- uses: actions/checkout@v4
23+
with:
24+
fetch-tags: true
25+
fetch-depth: 0
26+
- name: Mark repository as safe (Fix for https://github.com/actions/checkout/issues/1048)
27+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
28+
29+
- name: Prepare building packages
30+
run: |
31+
pip install build
32+
33+
- name: Build opencue_proto package
34+
id: build_opencue_proto
35+
run: |
36+
python -m build ./proto
37+
echo "opencue_proto_path=$(ls ./proto/dist/opencue_proto-*.whl)" >> $GITHUB_OUTPUT
38+
39+
- name: Build opencue_pycue package
40+
id: build_pycue
41+
run: |
42+
python -m build ./pycue
43+
echo "opencue_pycue_path=$(ls ./pycue/dist/opencue_pycue-*.whl)" >> $GITHUB_OUTPUT
44+
45+
- name: Build opencue_pyoutline package
46+
id: build_pyoutline
47+
run: |
48+
python -m build ./pyoutline
49+
echo "opencue_pyoutline_path=$(ls ./pyoutline/dist/opencue_pyoutline-*.whl)" >> $GITHUB_OUTPUT
50+
51+
- name: Build opencue_cueadmin package
52+
id: build_cueadmin
53+
run: |
54+
python -m build ./cueadmin
55+
echo "opencue_cueadmin_path=$(ls ./cueadmin/dist/opencue_cueadmin-*.whl)" >> $GITHUB_OUTPUT
56+
57+
- name: Build opencue_cuesubmit package
58+
id: build_cuesubmit
59+
run: |
60+
python -m build ./cuesubmit
61+
echo "opencue_cuesubmit_path=$(ls ./cuesubmit/dist/opencue_cuesubmit-*.whl)" >> $GITHUB_OUTPUT
62+
63+
- name: Build opencue_rqd package
64+
id: build_rqd
65+
run: |
66+
python -m build ./rqd
67+
echo "opencue_rqd_path=$(ls ./rqd/dist/opencue_rqd-*.whl)" >> $GITHUB_OUTPUT
68+
69+
- name: Upload opencue packages
70+
uses: actions/upload-artifact@v4
71+
with:
72+
name: opencue_packages
73+
path: |
74+
proto/dist/*.*
75+
pycue/dist/*.*
76+
pyoutline/dist/*.*
77+
cueadmin/dist/*.*
78+
cuesubmit/dist/*.*
79+
rqd/dist/*.*
80+
81+
install_opencue_packages_3_12:
82+
needs: build_opencue_packages
83+
name: Test installing packages with python 3.12
84+
runs-on: ubuntu-22.04
85+
container: python:3.12
86+
steps:
87+
- name: Download artifact
88+
uses: actions/download-artifact@v4
89+
with:
90+
name: opencue_packages
91+
- name: Install package
92+
run: |
93+
export OPENCUE_PROTO_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_proto_path }}"
94+
export OPENCUE_PYCUE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pycue_path }}"
95+
export OPENCUE_PYOUTLINE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pyoutline_path }}"
96+
export OPENCUE_CUEADMIN_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cueadmin_path }}"
97+
export OPENCUE_CUESUBMIT_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cuesubmit_path }}"
98+
export OPENCUE_RQD_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_rqd_path }}"
99+
pip install $OPENCUE_PROTO_PACKAGE_PATH $OPENCUE_PYCUE_PACKAGE_PATH $OPENCUE_PYOUTLINE_PACKAGE_PATH $OPENCUE_CUEADMIN_PACKAGE_PATH $OPENCUE_CUESUBMIT_PACKAGE_PATH $OPENCUE_RQD_PACKAGE_PATH
100+
101+
install_opencue_packages_3_11:
102+
needs: build_opencue_packages
103+
name: Test installing packages with python 3.11
104+
runs-on: ubuntu-22.04
105+
container: python:3.11
106+
steps:
107+
- name: Download a single artifact
108+
uses: actions/download-artifact@v4
109+
with:
110+
name: opencue_packages
111+
- name: Install package
112+
run: |
113+
export OPENCUE_PROTO_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_proto_path }}"
114+
export OPENCUE_PYCUE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pycue_path }}"
115+
export OPENCUE_PYOUTLINE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pyoutline_path }}"
116+
export OPENCUE_CUEADMIN_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cueadmin_path }}"
117+
export OPENCUE_CUESUBMIT_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cuesubmit_path }}"
118+
export OPENCUE_RQD_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_rqd_path }}"
119+
pip install $OPENCUE_PROTO_PACKAGE_PATH $OPENCUE_PYCUE_PACKAGE_PATH $OPENCUE_PYOUTLINE_PACKAGE_PATH $OPENCUE_CUEADMIN_PACKAGE_PATH $OPENCUE_CUESUBMIT_PACKAGE_PATH $OPENCUE_RQD_PACKAGE_PATH
120+
10121
test_python_2023:
122+
needs: build_opencue_packages
11123
name: Run Python Unit Tests (CY2023)
12124
runs-on: ubuntu-22.04
13125
container: aswf/ci-opencue:2023
14126
steps:
15127
- uses: actions/checkout@v4
128+
with:
129+
fetch-depth: 0
130+
fetch-tags: true
131+
- name: Mark repository as safe (Fix for https://github.com/actions/checkout/issues/1048)
132+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
133+
- name: Download a single artifact
134+
uses: actions/download-artifact@v4
135+
with:
136+
name: opencue_packages
16137
- name: Run Python Tests
17-
run: ci/run_python_tests.sh
138+
run: |
139+
export OPENCUE_PROTO_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_proto_path }}"
140+
export OPENCUE_PYCUE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pycue_path }}"
141+
export OPENCUE_PYOUTLINE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pyoutline_path }}"
142+
export OPENCUE_CUEADMIN_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cueadmin_path }}"
143+
export OPENCUE_CUESUBMIT_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cuesubmit_path }}"
144+
export OPENCUE_RQD_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_rqd_path }}"
145+
ci/run_python_tests.sh
18146
19147
test_cuebot_2023:
148+
needs: build_opencue_packages
20149
name: Build Cuebot and Run Unit Tests (CY2023)
21150
runs-on: ubuntu-22.04
22151
container:
@@ -29,15 +158,34 @@ jobs:
29158
su -c "cd cuebot && ./gradlew build --stacktrace --info" aswfuser
30159
31160
test_python_2024:
161+
needs: build_opencue_packages
32162
name: Run Python Unit Tests (CY2024)
33163
runs-on: ubuntu-22.04
34164
container: aswf/ci-opencue:2024
35165
steps:
36-
- uses: actions/checkout@v4
166+
- name: Checkout
167+
uses: actions/checkout@v4
168+
with:
169+
fetch-depth: 0
170+
fetch-tags: true
171+
- name: Mark repository as safe (Fix for https://github.com/actions/checkout/issues/1048)
172+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
173+
- name: Download a single artifact
174+
uses: actions/download-artifact@v4
175+
with:
176+
name: opencue_packages
37177
- name: Run Python Tests
38-
run: ci/run_python_tests.sh
178+
run: |
179+
export OPENCUE_PROTO_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_proto_path }}"
180+
export OPENCUE_PYCUE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pycue_path }}"
181+
export OPENCUE_PYOUTLINE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pyoutline_path }}"
182+
export OPENCUE_CUEADMIN_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cueadmin_path }}"
183+
export OPENCUE_CUESUBMIT_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cuesubmit_path }}"
184+
export OPENCUE_RQD_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_rqd_path }}"
185+
ci/run_python_tests.sh
39186
40187
test_cuebot_2024:
188+
needs: build_opencue_packages
41189
name: Build Cuebot and Run Unit Tests (CY2024)
42190
runs-on: ubuntu-22.04
43191
container:
@@ -50,14 +198,30 @@ jobs:
50198
su -c "cd cuebot && ./gradlew build --stacktrace --info" aswfuser
51199
52200
integration_test:
201+
needs: build_opencue_packages
53202
name: Run Integration Test
54203
runs-on: ubuntu-22.04
55204
steps:
56205
- name: Checkout
57206
uses: actions/checkout@v4
58-
207+
with:
208+
fetch-depth: 0
209+
fetch-tags: true
210+
- name: Mark repository as safe (Fix for https://github.com/actions/checkout/issues/1048)
211+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
212+
- name: Download a single artifact
213+
uses: actions/download-artifact@v4
214+
with:
215+
name: opencue_packages
59216
- name: Run test
60-
run: ci/run_integration_test.sh
217+
run: |
218+
export OPENCUE_PROTO_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_proto_path }}"
219+
export OPENCUE_PYCUE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pycue_path }}"
220+
export OPENCUE_PYOUTLINE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pyoutline_path }}"
221+
export OPENCUE_CUEADMIN_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cueadmin_path }}"
222+
export OPENCUE_CUESUBMIT_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cuesubmit_path }}"
223+
export OPENCUE_RQD_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_rqd_path }}"
224+
ci/run_integration_test.sh
61225
62226
- name: Archive log files
63227
uses: actions/upload-artifact@v4
@@ -67,23 +231,46 @@ jobs:
67231
path: /tmp/opencue-test/*.log
68232

69233
lint_python:
234+
needs: build_opencue_packages
70235
name: Lint Python Code
71236
runs-on: ubuntu-22.04
72237
container: aswf/ci-opencue:2024.1
73-
env:
74-
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
75238
steps:
76-
- uses: actions/checkout@v4
239+
- name: Checkout
240+
uses: actions/checkout@v4
241+
with:
242+
fetch-depth: 0
243+
fetch-tags: true
244+
- name: Mark repository as safe (Fix for https://github.com/actions/checkout/issues/1048)
245+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
246+
- name: Download a single artifact
247+
uses: actions/download-artifact@v4
248+
with:
249+
name: opencue_packages
250+
- name: Set package vars from parent action
251+
run: |
252+
export OPENCUE_PROTO_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_proto_path }}"
253+
export OPENCUE_PYCUE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pycue_path }}"
254+
export OPENCUE_PYOUTLINE_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_pyoutline_path }}"
255+
export OPENCUE_CUEADMIN_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cueadmin_path }}"
256+
export OPENCUE_CUESUBMIT_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_cuesubmit_path }}"
257+
export OPENCUE_RQD_PACKAGE_PATH="${{ needs.build_opencue_packages.outputs.opencue_rqd_path }}"
77258
- name: Lint Python Code
78259
run: ci/run_python_lint.sh
79260

80261
test_sphinx:
262+
needs: build_opencue_packages
81263
name: Test Documentation Build
82264
runs-on: ubuntu-22.04
83265
container:
84266
image: aswf/ci-opencue:2023
85267
steps:
86268
- uses: actions/checkout@v4
269+
with:
270+
fetch-depth: 0
271+
fetch-tags: true
272+
- name: Mark repository as safe (Fix for https://github.com/actions/checkout/issues/1048)
273+
run: git config --global --add safe.directory $GITHUB_WORKSPACE
87274
- name: Run Sphinx build
88275
run: ci/build_sphinx_docs.sh
89276

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,4 @@ rest_gateway/*.tar\.gz
2323
cuebot/.settings/*
2424
cuebot/.classpath
2525
cuebot/.project
26+
/proto/opencue_proto/

‎VERSION.in‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
1.8
1+
1.9

‎api_docs/conf.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
import os
1616
import sphinx_rtd_theme
1717
import sys
18+
sys.path.insert(0, os.path.abspath('../opencue_proto'))
1819
sys.path.insert(0, os.path.abspath('../pycue'))
1920
sys.path.insert(0, os.path.abspath('../pyoutline'))
2021

0 commit comments

Comments
 (0)