Skip to content

Commit 369fcd9

Browse files
author
bshifaw
authored
Add integration tests with Github Actions! (#172)
* Added new integration tests that run against a local cromwell instance running on Github actions!
1 parent 847d44e commit 369fcd9

File tree

15 files changed

+346
-45
lines changed

15 files changed

+346
-45
lines changed
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
name: Integration Tests
2+
3+
# Controls when the action will run.
4+
# Triggers the workflow on push or pull request events
5+
on: [push, workflow_dispatch]
6+
7+
jobs:
8+
build:
9+
runs-on: ubuntu-latest
10+
strategy:
11+
matrix:
12+
python: [ 3.7, 3.8, 3.9 ]
13+
14+
steps:
15+
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
16+
- uses: actions/checkout@v2
17+
# Using Conda to install womtool causes the default system python version
18+
# to be set to a later version. Placing the Install Womtool step before
19+
# Setup Python allows the tox test to run on py version specified in git matrix.
20+
- name: Install Womtool
21+
run: |
22+
$CONDA/bin/conda install -y -c bioconda womtool
23+
echo "$CONDA/bin" >> $GITHUB_PATH
24+
25+
- name: Setup Python
26+
uses: actions/setup-python@v2
27+
with:
28+
python-version: ${{ matrix.python }}
29+
30+
- name: Install Dev Requirements
31+
run: |
32+
python -m pip install --upgrade pip
33+
pip install -r dev-requirements.txt
34+
35+
- name: Pull and setup mysql dockers
36+
# Parameters set in the command below must be reused in the cromwell
37+
# config file: tests/resources/cromwell_application.conf
38+
run: |
39+
docker run \
40+
-p 3306:3306 \
41+
--ip=172.17.0.2 \
42+
--name NameOfTheContainer \
43+
-e MYSQL_ROOT_PASSWORD=YourPassword \
44+
-e MYSQL_DATABASE=DatabaseName \
45+
-e MYSQL_USER=ChooseAName \
46+
-e MYSQL_PASSWORD=YourOtherPassword \
47+
-d \
48+
mysql/mysql-server:5.5
49+
50+
- name: Pull and setup cromwell docker
51+
run: |
52+
export CROMWELL_TAG=$(curl -s https://api.github.com/repos/broadinstitute/cromwell/releases/latest | \
53+
grep -o '"tag_name": "[^"]*' | \
54+
grep -o '[^"]*$' | \
55+
grep -Eo "^[0-9]+")
56+
57+
docker run \
58+
-p 8000:8000 \
59+
-v ${PWD}/tests/resources/cromwell_application.conf:/cromwell_application.conf \
60+
--name CromwellContainer \
61+
-e JAVA_OPTS="-Dconfig.file=/cromwell_application.conf" \
62+
-d broadinstitute/cromwell:${CROMWELL_TAG} server
63+
64+
# Give cromwell server time to setup
65+
sleep 1m
66+
67+
- name: Install Tox
68+
run: pip install tox
69+
70+
- name: Run Tox integration tests
71+
# Run tox using the version of Python in `PATH`
72+
run: tox -e integration

.github/workflows/main.yml renamed to .github/workflows/unit_tests.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: CI
1+
name: Unit Tests
22

33
# Controls when the action will run.
44
# Triggers the workflow on push or pull request events
@@ -12,7 +12,7 @@ jobs:
1212
runs-on: ubuntu-latest
1313
strategy:
1414
matrix:
15-
python: [ 3.6, 3.7, 3.8 ]
15+
python: [ 3.7, 3.8, 3.9 ]
1616

1717

1818
# Steps represent a sequence of tasks that will be executed as part of the job

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,8 @@ Current version: 2.0.0
9393
It will copy your wdl and json inputs into the folder for reproducibility.
9494
* It keeps track of your most recently submitted jobs by storing their ids in `./cromshell/`
9595
You may omit the job ID of the last job submitted when running commands, or use negative numbers to reference previous jobs, e.g. "-1" will track the last job, "-2" will track the one before that, and so on.
96-
* You can override the default cromwell server by setting the environmental variable `CROMWELL_URL` to the appropriate URL.
96+
* You can override the default cromwell server by setting the argument `--cromwell_url` to the appropriate URL.
97+
* You can override the default cromshell configuration folder by setting the environmental variable `CROMSHELL_DIR` to the appropriate directory.
9798
* Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `./cromwell status -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`).
9899

99100
## Installation

setup.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,14 @@
1717
install_requires="""
1818
dataclasses
1919
termcolor
20-
click
20+
click>=8.0.0
2121
requests
2222
pygments
2323
""".split(
2424
"\n"
2525
),
2626
tests_require=["coverage", "pytest"],
27-
python_requires=">=3.6",
27+
python_requires=">=3.7",
2828
packages=find_packages("src"),
2929
package_dir={"": "src"},
3030
classifiers=[
@@ -33,9 +33,9 @@
3333
"License :: OSI Approved :: BSD 3-Clause",
3434
"Natural Language :: English",
3535
"Operating System :: OS Independent",
36-
"Programming Language :: Python :: 3.6",
3736
"Programming Language :: Python :: 3.7",
3837
"Programming Language :: Python :: 3.8",
38+
"Programming Language :: Python :: 3.9",
3939
"Programming Language :: Python :: Implementation :: CPython",
4040
],
4141
entry_points={"console_scripts": ["cromshell=cromshell.__main__:main_entry"]},

src/cromshell/__main__.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,14 @@ def main_entry(
7373
requests_timeout,
7474
requests_skip_certs,
7575
):
76+
"""
77+
Cromshell is a script for submitting workflows to a
78+
cromwell server and monitoring / querying their results.\n
79+
Notes:\n
80+
- A hidden folder will be created on initial run. The hidden folder
81+
(.../.cromshell) will be placed in users home directory but can be overridden
82+
by setting environment variable 'CROMSHELL_CONFIG'.
83+
"""
7684
# Set up our log verbosity
7785
from . import log # pylint: disable=C0415
7886

src/cromshell/log.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ def configure_logging(verbosity):
3333
# Set logging level:
3434
log_level = logging.INFO
3535
if verbosity:
36-
log_level = verbosity
36+
log_level = int(verbosity)
3737

3838
# Create our basic config:
3939
logging.basicConfig(level=log_level, format=format_string)

src/cromshell/submit/command.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,8 +105,9 @@ def main(config, wdl, wdl_json, options_json, dependencies_zip):
105105
# TODO: Refactor these file manipulations into its own "cleanup" function?
106106
# If we get here, we successfully submitted the job and should track it locally:
107107
# 1. Create a directory to hold function input files, using server name
108+
server_folder_name = config.get_local_folder_name()
108109
run_directory = Path(config.config_dir).joinpath(
109-
config.local_folder_name, workflow_status["id"]
110+
server_folder_name, workflow_status["id"]
110111
)
111112
io_utils.create_directory(run_directory)
112113

src/cromshell/utilities/cromshellconfig.py

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,11 @@
2727
# Defaults for variables will be set after functions have been defined
2828
config_dir = None
2929
SUBMISSION_FILE_NAME = "all.workflow.database.tsv"
30-
SUBMISSION_FILE_HEADER = "DATE\tCROMWELL_SERVER\tRUN_ID\tWDL_NAME\tSTATUS\tALIAS"
30+
SUBMISSION_FILE_HEADER = "DATE\tCROMWELL_SERVER\tRUN_ID\tWDL_NAME\tSTATUS\tALIAS\n"
3131
CROMSHELL_CONFIG_FILE_NAME = "cromshell_config.json"
3232
submission_file_path = None
3333
cromshell_config_options = None
3434
cromwell_server = None
35-
local_folder_name = None
3635
# Request defaults
3736
requests_connect_timeout = 5
3837
requests_verify_certs = True
@@ -125,8 +124,20 @@ def resolve_cromwell_config_server_address(server_user=None, workflow_id=None):
125124
def __get_config_dir():
126125
"""Get Path To Cromshell Hidden Directory"""
127126

128-
config_path = os.path.join(Path.home(), ".cromshell")
129-
Path.mkdir(Path(config_path), exist_ok=True)
127+
# If env CROMSHELL_CONFIG set then use for cromshell hidden dir else use home dir.
128+
if os.environ.get("CROMSHELL_CONFIG"):
129+
LOGGER.info(
130+
"Detected 'CROMSHELL_CONFIG' in environment, using {config_path} as "
131+
"cromshell hidden directory."
132+
)
133+
config_path = os.path.join(os.environ.get("CROMSHELL_CONFIG"), ".cromshell")
134+
135+
else:
136+
config_path = os.path.join(Path.home(), ".cromshell")
137+
138+
Path.mkdir(Path(config_path), exist_ok=True, parents=True)
139+
LOGGER.info(f"Cromshell config directory set to {config_path}.")
140+
130141
return config_path
131142

132143

@@ -181,6 +192,11 @@ def get_cromwell_api():
181192
return f"{cromwell_server}{API_STRING}"
182193

183194

195+
def get_local_folder_name():
196+
"""Return a string combining the cromwell server without http/https"""
197+
return cromwell_server.replace("https://", "").replace("http://", "")
198+
199+
184200
def resolve_requests_connect_timeout(timeout_cli: int):
185201
"""Override the default request timeout duration.
186202
@@ -220,4 +236,3 @@ def resolve_requests_connect_timeout(timeout_cli: int):
220236
config_dir, CROMSHELL_CONFIG_FILE_NAME, CROMSHELL_CONFIG_OPTIONS_TEMPLATE
221237
)
222238
cromwell_server = __get_cromwell_server(cromshell_config_options)
223-
local_folder_name = cromwell_server.replace("https://", "").replace("http://", "")

test-requirements.txt

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
11
pytest
22
pytest-cov
3+
pytest-dependency
34
coverage >= 4.5
5+
click>=8.0.0
6+
requests
7+
termcolor
8+
requests
9+
pygments

tests/integration/test_placeholder.py

Lines changed: 0 additions & 22 deletions
This file was deleted.

0 commit comments

Comments
 (0)