Skip to content

Use of new taxonomy tools and introduction of postinstall django app commands management #11136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nastasi-oq wants to merge 19 commits into
base: engine-3.23
Choose a base branch
from
Open

Use of new taxonomy tools and introduction of postinstall django app commands management #11136

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
6e332fa
update URL_STANDALONE for b3.23-pre5 branch
nastasi-oq Dec 17, 2025
f97eeca
set STANDALONE_APP_NAME_MAP in settings.py directly
nastasi-oq Dec 24, 2025
6dc82aa
add 'django-gem-taxonomy' as new standalone app
nastasi-oq Dec 24, 2025
764d14a
manage properly 'openquake' namespace: remove 'openquake/__init__.py'…
nastasi-oq Jan 19, 2026
98df5f5
introduce post installation command management for standalone applica…
nastasi-oq Jan 19, 2026
7d6f24a
some cleanup
nastasi-oq Jan 20, 2026
5c26159
differentiate 'django-admin' command for different OS
nastasi-oq Jan 20, 2026
1740d83
modify environment to be able to run django-admin with the proper set…
nastasi-oq Jan 20, 2026
93a2a0f
wheelhouse link to engine-3.23 folder
nastasi-oq Jan 21, 2026
0c9140b
more clear if condition
nastasi-oq Jan 21, 2026
e681d0b
actualize template before run oq-engine install script to avoid error…
nastasi-oq Jan 21, 2026
d94714d
add '--keepvenv' argument to 'install.py' to preserve previous virtua…
nastasi-oq Jan 22, 2026
a0795da
from keepvenv to no_env argument
nastasi-oq Jan 22, 2026
925a0e8
manage VENV path for no_venv case
nastasi-oq Jan 22, 2026
313847f
rename install argument
nastasi-oq Jan 22, 2026
153e6e1
add '--noupgrade' argument to install.py command
nastasi-oq Jan 23, 2026
d7808fd
wrong default argparse default values fixed
nastasi-oq Jan 23, 2026
947e2d4
fix operators precedence order
nastasi-oq Jan 23, 2026
9b0bd35
add a docker_latest workflow to be able to run it in the current gith…
nastasi-oq Jan 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/workflows/docker_latest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
name: Build Image Latest for Docker
on:
workflow_dispatch:
inputs:
tags:
description: Comma-separated list of tags
default: latest
required: true
git-ref:
description: Git Ref
default: master
required: true

jobs:
docker:
name: Build image and push after successfull calculation
runs-on: ubuntu-latest

steps:
# This Checkout is necessary when using a context in docker/build-push-action
- name: Clone Repository (Latest)
uses: actions/checkout@v4
if: github.event.inputs.git-ref == ''
- name: Clone Repository (Custom Ref)
uses: actions/checkout@v4
if: github.event.inputs.git-ref != ''
with:
ref: ${{ github.event.inputs.git-ref }}
- name: Extract tag names
shell: bash
run: echo "##[set-output name=tags;]" | tr -d '\n'; for tag in $(echo ${{ github.event.inputs.tags }} | tr , '\n'); do echo "-t openquake/engine:$tag " | tr -d '\n'; done
id: extract_tags
- name: Build image engine with tag ${{ github.event.inputs.version }}
env:
DOCKER_USERNAME: ${{ secrets.docker_username }}
DOCKER_PASSWORD: ${{ secrets.docker_password }}
REPO_REF: ${{ github.event.inputs.git-ref }}
id: docker_engine
run: docker build --build-arg oq_branch=$REPO_REF ${{ steps.extract_tags.outputs.tags }} -f docker/Dockerfile.engine docker
- name: List Image
run: |
docker image ls
- name: Run calcs on single docker
run: |
time docker run openquake/engine:latest "oq engine --run "https://github.com/gem/oq-engine/blob/master/openquake/server/tests/data/classical.zip?raw=true""
- name: push image engine with tags ${{ github.event.inputs.tags }} on dockerhub
env:
DOCKER_USERNAME: ${{ secrets.docker_username }}
DOCKER_PASSWORD: ${{ secrets.docker_password }}
DOCKER_TAG: ${{ github.event.inputs.version }}
run: |
docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"
docker push openquake/engine --all-tags
20 changes: 10 additions & 10 deletions .github/workflows/engine_pr_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,18 +160,18 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Actualize 'aelo' templates for email notifications
run: |
for file in openquake/server/templates/registration/*.aelo.tmpl; do
cp -- "$file" "${file%.aelo.tmpl}"
done
- name: Install dependencies
run: |
if [[ $FROM_FORK == "true" ]]; then
python install.py devel --version=$GITHUB_HEAD_REF --from_fork
else
python install.py devel --version=$GITHUB_HEAD_REF
fi
- name: Actualize 'aelo' templates for email notifications
run: |
for file in openquake/server/templates/registration/*.aelo.tmpl; do
cp -- "$file" "${file%.aelo.tmpl}"
done
- name: Server 'AELO' mode tests
run: |
source ~/openquake/bin/activate
Expand Down Expand Up @@ -199,18 +199,18 @@ jobs:
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Actualize 'impact' templates for email notifications
run: |
for file in openquake/server/templates/registration/*.impact.tmpl; do
cp -- "$file" "${file%.impact.tmpl}"
done
- name: Install dependencies
run: |
if [[ $FROM_FORK == "true" ]]; then
python install.py devel --version=$GITHUB_HEAD_REF --from_fork
else
python install.py devel --version=$GITHUB_HEAD_REF
fi
- name: Actualize 'impact' templates for email notifications
run: |
for file in openquake/server/templates/registration/*.impact.tmpl; do
cp -- "$file" "${file%.impact.tmpl}"
done
- name: Server 'ARISTOTLE' mode tests
run: |
set -x
Expand Down
144 changes: 99 additions & 45 deletions install.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ class devel(user):
}
DEMOS = "https://artifacts.openquake.org/travis/demos-master.zip"
GITBRANCH = "https://github.com/gem/oq-engine/archive/%s.zip"
URL_STANDALONE = "https://wheelhouse.openquake.org/py/standalone/latest/"
URL_STANDALONE = "https://wheelhouse.openquake.org/py/standalone/engine-3.23/"


def ensure(pip=None, pyvenv=None):
Expand Down Expand Up @@ -263,38 +263,76 @@ def get_requirements_branch(version, inst, from_fork):
return version


def install_standalone(venv):
def install_or_postinstall_standalone(venv, is_install=True):
"""
Copy link
Contributor

@micheles micheles Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename is_install as installed

Install the standalone Django applications if possible
Install the standalone Django applications if possible or
run '<app>_postinstall' command if it exists
"""
errors = []
print("The standalone applications are not installed yet")
if is_install:
print("The standalone applications are not installed yet")
else:
print("Run '<app>_postinstall' command for each standalone\n"
" Django applications, if it exists")
if sys.platform == "win32":
if os.path.exists("python\\python._pth.old"):
pycmd = inst.VENV + "\\python.exe"
else:
pycmd = inst.VENV + "\\Scripts\\python.exe"
else:
pycmd = inst.VENV + "/bin/python3"
for app in [
"oq-platform-standalone",
"oq-platform-ipt",
"oq-platform-taxonomy",
]:
try:
print("Applications " + app + " are not installed yet \n")

subprocess.check_call(
[pycmd, "-m", "pip", "install", "--find-links", URL_STANDALONE,
app]
)
except Exception as exc:
# for instance is somebody removed a wheel from the wheelhouse
errors.append("%s: could not install %s" % (exc, app))
STANDALONE_APP_INFO = [
{"pkg": "oq-platform-standalone", "name": None},
{"pkg": "oq-platform-ipt", "name": "openquakeplatform_ipt"},
{"pkg": "oq-platform-taxonomy", "name": "openquakeplatform_taxonomy"},
{"pkg": "django-gem-taxonomy", "name": "django_gem_taxonomy"},
]

if is_install:
for app in STANDALONE_APP_INFO:
try:
print("Applications " + app['pkg'] + " are not installed yet \n")

subprocess.check_call(
[pycmd, "-m", "pip", "install", "--find-links", URL_STANDALONE,
app['pkg']]
)
except Exception as exc:
# for instance is somebody removed a wheel from the wheelhouse
errors.append("%s: could not install %s" % (exc, app['pkg']))
else:
for app in STANDALONE_APP_INFO:
if not app['name']:
continue

try:
if sys.platform == "win32":
django_admin = ['Scripts', 'django-admin.exe']
else:
django_admin = ["bin", "django-admin"]

django_env = os.environ.copy()
django_env["DJANGO_SETTINGS_MODULE"] = "openquake.server.settings"

subprocess.check_call(
[os.path.join(inst.VENV, *django_admin),
"openquake_engine_postinstall", app['name']],
env=django_env)
except Exception as exc:
# for instance is somebody removed a wheel from the wheelhouse
errors.append("%s: error during %s postinstall command execution" % (exc, app['name']))

return errors


def before_checks(inst, venv, port, remove, usage):
def install_standalone(venv):
return install_or_postinstall_standalone(venv, is_install=True)

def postinstall_standalone(venv):
return install_or_postinstall_standalone(venv, is_install=False)

def before_checks(inst, venv, port, remove, novenv, usage):
"""
Checks to perform before the installation
"""
Expand All @@ -303,6 +341,10 @@ def before_checks(inst, venv, port, remove, usage):
if port:
inst.DBPORT = int(port)

if novenv:
inst.VENV = os.path.join(os.getenv('LocalAppData'), 'Programs',
'OpenQuake Engine', 'python3')

# check platform
if (inst is server and sys.platform != "linux") or (
inst is devel_server and sys.platform != "linux"
Expand Down Expand Up @@ -386,7 +428,7 @@ def fix_version(commit, venv):
f.write("".join(lines))


def install(inst, version, from_fork):
def install(inst, version, from_fork, novenv, noupgrade):
"""
Install the engine in one of the three possible modes
"""
Expand All @@ -406,34 +448,38 @@ def install(inst, version, from_fork):
if inst is server or inst is devel_server:
subprocess.check_call(["chown", "openquake", inst.OQDATA])

# recreate the openquake venv
ensure(pyvenv=inst.VENV)
print("Created %s" % inst.VENV)
if not novenv:
# recreate the openquake venv
ensure(pyvenv=inst.VENV)
print("Created %s" % inst.VENV)

if sys.platform == "win32":
if os.path.exists("python\\python._pth.old"):
pycmd = inst.VENV + "\\python.exe"
if sys.platform == "win32":
if os.path.exists("python\\python._pth.old"):
pycmd = inst.VENV + "\\python.exe"
else:
pycmd = inst.VENV + "\\Scripts\\python.exe"
else:
pycmd = inst.VENV + "\\Scripts\\python.exe"
pycmd = inst.VENV + "/bin/python3"
else:
pycmd = inst.VENV + "/bin/python3"

pycmd = os.path.join(inst.VENV, 'python.exe')
# upgrade pip and before check that it is installed in venv
if sys.platform != "win32":
ensure(pip=pycmd)
subprocess.check_call(
[pycmd, "-m", "pip", "install", "--upgrade", "pip", "wheel"]
)
subprocess.check_call([pycmd, "-m", "pip", "install"] + ([
] if noupgrade else ["--upgrade"]) + [
"pip", "wheel"])
else:
if os.path.exists("python\\python._pth.old"):
subprocess.check_call(
[pycmd, "-m", "pip", "install", "--upgrade", "pip", "wheel",
"urllib3"])
subprocess.check_call([pycmd, "-m", "pip", "install"] + ([
] if noupgrade else ["--upgrade"]) + [
"pip", "wheel", "urllib3"])
else:
subprocess.check_call([pycmd, "-m", "ensurepip", "--upgrade"])
subprocess.check_call(
[pycmd, "-m", "pip", "install", "--upgrade", "pip", "wheel",
"urllib3"])
subprocess.check_call([pycmd, "-m", "ensurepip"] + ([
] if noupgrade else ["--upgrade"]))

subprocess.check_call([pycmd, "-m", "pip", "install"] + ([
] if noupgrade else ["--upgrade"]) + [
"pip", "wheel", "urllib3"])

# install the requirements
branch = get_requirements_branch(version, inst, from_fork)
Expand Down Expand Up @@ -465,18 +511,20 @@ def install(inst, version, from_fork):
subprocess.check_call([pycmd, "-m", "pip", "install", "-e", CDIR])
elif version is None: # install the stable version
subprocess.check_call(
[pycmd, "-m", "pip", "install", "--upgrade", "openquake.engine"]
[pycmd, "-m", "pip", "install"] + ([
] if noupgrade else ["--upgrade"]) + ["openquake.engine"]
)
elif re.match(r"\d+(\.\d+)+", version): # install an official version
subprocess.check_call(
[pycmd, "-m", "pip", "install", "--upgrade",
"openquake.engine==" + version]
[pycmd, "-m", "pip", "install"] + ([
] if noupgrade else ["--upgrade"]) + ["openquake.engine==" + version]
)
else: # install a branch from github (only for user or server)
commit = latest_commit(version)
print("Installing commit", commit)
subprocess.check_call(
[pycmd, "-m", "pip", "install", "--upgrade", GITBRANCH % commit]
[pycmd, "-m", "pip", "install"] + ([
] if noupgrade else ["--upgrade"]) + [GITBRANCH % commit]
)
fix_version(commit, inst.VENV)

Expand Down Expand Up @@ -506,6 +554,8 @@ def install(inst, version, from_fork):
if inst in (user, devel): # create/upgrade the db in the default location
subprocess.run([oqreal, "engine", "--upgrade-db"])

errors += postinstall_standalone(inst.VENV)

if (
inst is server
and not os.path.exists(inst.OQ)
Expand Down Expand Up @@ -610,6 +660,10 @@ def remove(inst):
help="the kind of installation you want",
)
parser.add_argument("--venv", help="venv directory")
parser.add_argument("--novenv", action="store_true",
help="keep the current python environment")
parser.add_argument("--noupgrade", action="store_true",
help="not use '--upgrade' in pip install calls")
parser.add_argument("--remove", action="store_true",
help="disinstall the engine")
parser.add_argument("--version", help="version to install (default stable)")
Expand All @@ -623,12 +677,12 @@ def remove(inst):
args = parser.parse_args()
if args.inst:
inst = globals()[args.inst]
before_checks(inst, args.venv, args.dbport, args.remove,
before_checks(inst, args.venv, args.dbport, args.remove, args.novenv,
parser.format_usage())
if args.remove:
remove(inst)
else:
errors = install(inst, args.version, args.from_fork)
errors = install(inst, args.version, args.from_fork, args.novenv, args.noupgrade)
if errors:
# NB: even if one of the tools is missing, the engine will work
sys.exit('\n'.join(errors))
Expand Down
19 changes: 0 additions & 19 deletions openquake/__init__.py
View file
Open in desktop

This file was deleted.

Loading
Loading