Skip to content

Commit 369fcd9

Browse files
author
bshifaw
authored
Add integration tests with Github Actions! (#172)
* Added new integration tests that run against a local cromwell instance running on Github actions!
1 parent 847d44e commit 369fcd9

15 files changed

+346
-45
lines changed
+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
name: Integration Tests
2+
3+
# Controls when the action will run.
4+
# Triggers the workflow on push or pull request events
5+
on: [push, workflow_dispatch]
6+
7+
jobs:
8+
build:
9+
runs-on: ubuntu-latest
10+
strategy:
11+
matrix:
12+
python: [ 3.7, 3.8, 3.9 ]
13+
14+
steps:
15+
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
16+
- uses: actions/checkout@v2
17+
# Using Conda to install womtool causes the default system python version
18+
# to be set to a later version. Placing the Install Womtool step before
19+
# Setup Python allows the tox test to run on py version specified in git matrix.
20+
- name: Install Womtool
21+
run: |
22+
$CONDA/bin/conda install -y -c bioconda womtool
23+
echo "$CONDA/bin" >> $GITHUB_PATH
24+
25+
- name: Setup Python
26+
uses: actions/setup-python@v2
27+
with:
28+
python-version: ${{ matrix.python }}
29+
30+
- name: Install Dev Requirements
31+
run: |
32+
python -m pip install --upgrade pip
33+
pip install -r dev-requirements.txt
34+
35+
- name: Pull and setup mysql dockers
36+
# Parameters set in the command below must be reused in the cromwell
37+
# config file: tests/resources/cromwell_application.conf
38+
run: |
39+
docker run \
40+
-p 3306:3306 \
41+
--ip=172.17.0.2 \
42+
--name NameOfTheContainer \
43+
-e MYSQL_ROOT_PASSWORD=YourPassword \
44+
-e MYSQL_DATABASE=DatabaseName \
45+
-e MYSQL_USER=ChooseAName \
46+
-e MYSQL_PASSWORD=YourOtherPassword \
47+
-d \
48+
mysql/mysql-server:5.5
49+
50+
- name: Pull and setup cromwell docker
51+
run: |
52+
export CROMWELL_TAG=$(curl -s https://api.github.com/repos/broadinstitute/cromwell/releases/latest | \
53+
grep -o '"tag_name": "[^"]*' | \
54+
grep -o '[^"]*$' | \
55+
grep -Eo "^[0-9]+")
56+
57+
docker run \
58+
-p 8000:8000 \
59+
-v ${PWD}/tests/resources/cromwell_application.conf:/cromwell_application.conf \
60+
--name CromwellContainer \
61+
-e JAVA_OPTS="-Dconfig.file=/cromwell_application.conf" \
62+
-d broadinstitute/cromwell:${CROMWELL_TAG} server
63+
64+
# Give cromwell server time to setup
65+
sleep 1m
66+
67+
- name: Install Tox
68+
run: pip install tox
69+
70+
- name: Run Tox integration tests
71+
# Run tox using the version of Python in `PATH`
72+
run: tox -e integration

.github/workflows/main.yml renamed to .github/workflows/unit_tests.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: CI
1+
name: Unit Tests
22

33
# Controls when the action will run.
44
# Triggers the workflow on push or pull request events
@@ -12,7 +12,7 @@ jobs:
1212
runs-on: ubuntu-latest
1313
strategy:
1414
matrix:
15-
python: [ 3.6, 3.7, 3.8 ]
15+
python: [ 3.7, 3.8, 3.9 ]
1616

1717

1818
# Steps represent a sequence of tasks that will be executed as part of the job

README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,8 @@ Current version: 2.0.0
9393
It will copy your wdl and json inputs into the folder for reproducibility.
9494
* It keeps track of your most recently submitted jobs by storing their ids in `./cromshell/`
9595
You may omit the job ID of the last job submitted when running commands, or use negative numbers to reference previous jobs, e.g. "-1" will track the last job, "-2" will track the one before that, and so on.
96-
* You can override the default cromwell server by setting the environmental variable `CROMWELL_URL` to the appropriate URL.
96+
* You can override the default cromwell server by setting the argument `--cromwell_url` to the appropriate URL.
97+
* You can override the default cromshell configuration folder by setting the environmental variable `CROMSHELL_DIR` to the appropriate directory.
9798
* Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `./cromwell status -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`).
9899

99100
## Installation

setup.py

+3-3
Original file line numberDiff line numberDiff line change
@@ -17,14 +17,14 @@
1717
install_requires="""
1818
dataclasses
1919
termcolor
20-
click
20+
click>=8.0.0
2121
requests
2222
pygments
2323
""".split(
2424
"\n"
2525
),
2626
tests_require=["coverage", "pytest"],
27-
python_requires=">=3.6",
27+
python_requires=">=3.7",
2828
packages=find_packages("src"),
2929
package_dir={"": "src"},
3030
classifiers=[
@@ -33,9 +33,9 @@
3333
"License :: OSI Approved :: BSD 3-Clause",
3434
"Natural Language :: English",
3535
"Operating System :: OS Independent",
36-
"Programming Language :: Python :: 3.6",
3736
"Programming Language :: Python :: 3.7",
3837
"Programming Language :: Python :: 3.8",
38+
"Programming Language :: Python :: 3.9",
3939
"Programming Language :: Python :: Implementation :: CPython",
4040
],
4141
entry_points={"console_scripts": ["cromshell=cromshell.__main__:main_entry"]},

src/cromshell/__main__.py

+8
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,14 @@ def main_entry(
7373
requests_timeout,
7474
requests_skip_certs,
7575
):
76+
"""
77+
Cromshell is a script for submitting workflows to a
78+
cromwell server and monitoring / querying their results.\n
79+
Notes:\n
80+
- A hidden folder will be created on initial run. The hidden folder
81+
(.../.cromshell) will be placed in users home directory but can be overridden
82+
by setting environment variable 'CROMSHELL_CONFIG'.
83+
"""
7684
# Set up our log verbosity
7785
from . import log # pylint: disable=C0415
7886

src/cromshell/log.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ def configure_logging(verbosity):
3333
# Set logging level:
3434
log_level = logging.INFO
3535
if verbosity:
36-
log_level = verbosity
36+
log_level = int(verbosity)
3737

3838
# Create our basic config:
3939
logging.basicConfig(level=log_level, format=format_string)

src/cromshell/submit/command.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -105,8 +105,9 @@ def main(config, wdl, wdl_json, options_json, dependencies_zip):
105105
# TODO: Refactor these file manipulations into its own "cleanup" function?
106106
# If we get here, we successfully submitted the job and should track it locally:
107107
# 1. Create a directory to hold function input files, using server name
108+
server_folder_name = config.get_local_folder_name()
108109
run_directory = Path(config.config_dir).joinpath(
109-
config.local_folder_name, workflow_status["id"]
110+
server_folder_name, workflow_status["id"]
110111
)
111112
io_utils.create_directory(run_directory)
112113

src/cromshell/utilities/cromshellconfig.py

+20-5
Original file line numberDiff line numberDiff line change
@@ -27,12 +27,11 @@
2727
# Defaults for variables will be set after functions have been defined
2828
config_dir = None
2929
SUBMISSION_FILE_NAME = "all.workflow.database.tsv"
30-
SUBMISSION_FILE_HEADER = "DATE\tCROMWELL_SERVER\tRUN_ID\tWDL_NAME\tSTATUS\tALIAS"
30+
SUBMISSION_FILE_HEADER = "DATE\tCROMWELL_SERVER\tRUN_ID\tWDL_NAME\tSTATUS\tALIAS\n"
3131
CROMSHELL_CONFIG_FILE_NAME = "cromshell_config.json"
3232
submission_file_path = None
3333
cromshell_config_options = None
3434
cromwell_server = None
35-
local_folder_name = None
3635
# Request defaults
3736
requests_connect_timeout = 5
3837
requests_verify_certs = True
@@ -125,8 +124,20 @@ def resolve_cromwell_config_server_address(server_user=None, workflow_id=None):
125124
def __get_config_dir():
126125
"""Get Path To Cromshell Hidden Directory"""
127126

128-
config_path = os.path.join(Path.home(), ".cromshell")
129-
Path.mkdir(Path(config_path), exist_ok=True)
127+
# If env CROMSHELL_CONFIG set then use for cromshell hidden dir else use home dir.
128+
if os.environ.get("CROMSHELL_CONFIG"):
129+
LOGGER.info(
130+
"Detected 'CROMSHELL_CONFIG' in environment, using {config_path} as "
131+
"cromshell hidden directory."
132+
)
133+
config_path = os.path.join(os.environ.get("CROMSHELL_CONFIG"), ".cromshell")
134+
135+
else:
136+
config_path = os.path.join(Path.home(), ".cromshell")
137+
138+
Path.mkdir(Path(config_path), exist_ok=True, parents=True)
139+
LOGGER.info(f"Cromshell config directory set to {config_path}.")
140+
130141
return config_path
131142

132143

@@ -181,6 +192,11 @@ def get_cromwell_api():
181192
return f"{cromwell_server}{API_STRING}"
182193

183194

195+
def get_local_folder_name():
196+
"""Return a string combining the cromwell server without http/https"""
197+
return cromwell_server.replace("https://", "").replace("http://", "")
198+
199+
184200
def resolve_requests_connect_timeout(timeout_cli: int):
185201
"""Override the default request timeout duration.
186202
@@ -220,4 +236,3 @@ def resolve_requests_connect_timeout(timeout_cli: int):
220236
config_dir, CROMSHELL_CONFIG_FILE_NAME, CROMSHELL_CONFIG_OPTIONS_TEMPLATE
221237
)
222238
cromwell_server = __get_cromwell_server(cromshell_config_options)
223-
local_folder_name = cromwell_server.replace("https://", "").replace("http://", "")

test-requirements.txt

+6
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
11
pytest
22
pytest-cov
3+
pytest-dependency
34
coverage >= 4.5
5+
click>=8.0.0
6+
requests
7+
termcolor
8+
requests
9+
pygments

tests/integration/test_placeholder.py

-22
This file was deleted.

tests/integration/test_submit.py

+117
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
import json
2+
import os
3+
from pathlib import Path
4+
from traceback import print_exception
5+
6+
import pytest
7+
from click.testing import CliRunner
8+
9+
from cromshell.__main__ import main_entry as cromshell
10+
11+
12+
def run_cromshell_submit(
13+
wdl: str, json_file: str, local_cromwell_url: str, exit_code: int
14+
):
15+
"""Run cromshell submit using CliRunner and assert job is successful"""
16+
17+
runner = CliRunner(mix_stderr=False)
18+
# The absolute path will be passed to the invoke command because
19+
# the test is being run in temp directory created by CliRunner.
20+
absolute_wdl = str(Path(wdl).resolve())
21+
absolute_json = str(Path(json_file).resolve())
22+
with runner.isolated_filesystem():
23+
result = runner.invoke(
24+
cromshell,
25+
[
26+
"--cromwell_url",
27+
local_cromwell_url,
28+
"--hide_logo",
29+
"submit",
30+
absolute_wdl,
31+
absolute_json,
32+
],
33+
)
34+
assert result.exit_code == exit_code, (
35+
f"\nSTDOUT:\n{result.stdout}"
36+
f"\nSTDERR:\n{result.stderr}"
37+
f"\nExceptions:\n{result.exception}"
38+
f"\n{print_exception(*result.exc_info)}"
39+
)
40+
return result
41+
42+
43+
def workflow_id_in_txt_db(result, local_workflow_database_tsv: Path):
44+
"""Get workflow id and assert it is listed in the local workflow database tsv"""
45+
46+
stdout_substring_formatted = json.loads(result.stdout)
47+
test_workflow_id = stdout_substring_formatted["id"]
48+
49+
# check if workflow id is in database tsv
50+
with open(local_workflow_database_tsv, "r") as file1:
51+
52+
# read file content
53+
readfile = file1.read()
54+
# checking condition for string found or not
55+
assert (
56+
test_workflow_id in readfile
57+
), "Workflow ID was not found in /all.workflow.database.tsv"
58+
59+
60+
class TestSubmit:
61+
@pytest.mark.dependency(name="test_submit")
62+
@pytest.mark.parametrize(
63+
"wdl, json_file, exit_code",
64+
[
65+
("tests/workflows/helloWorld.wdl", "tests/workflows/helloWorld.json", 0),
66+
("tests/workflows/helloWorld.wdl", "tests/workflows/not_valid.json", 1),
67+
("tests/workflows/not_valid.wdl", "tests/workflows/helloWorld.json", 1),
68+
],
69+
)
70+
def test_submit(
71+
self,
72+
local_cromwell_url: str,
73+
wdl: str,
74+
json_file: str,
75+
exit_code: int,
76+
local_hidden_cromshell_folder: Path,
77+
local_workflow_database_tsv: Path,
78+
):
79+
# Run cromshell submit
80+
result = run_cromshell_submit(
81+
wdl=wdl,
82+
json_file=json_file,
83+
exit_code=exit_code,
84+
local_cromwell_url=local_cromwell_url,
85+
)
86+
87+
# If submission passed check workflow id in database tsv
88+
if exit_code == 0:
89+
workflow_id_in_txt_db(
90+
result=result, local_workflow_database_tsv=local_workflow_database_tsv
91+
)
92+
93+
@pytest.mark.dependency(depends=["test_submit"])
94+
def test_submit_cromshell_folders_created(
95+
self, local_server_folder, local_hidden_cromshell_folder
96+
):
97+
# Created hidden cromshell directory
98+
assert Path.exists(local_hidden_cromshell_folder)
99+
# Created a directory to hold function input files, using server name
100+
assert Path.exists(local_server_folder)
101+
102+
@pytest.fixture
103+
def local_cromwell_url(self):
104+
return "http://localhost:8000"
105+
106+
@pytest.fixture
107+
def local_hidden_cromshell_folder(self):
108+
return Path(os.environ.get("CROMSHELL_CONFIG")).joinpath(".cromshell")
109+
110+
@pytest.fixture
111+
def local_workflow_database_tsv(self, local_hidden_cromshell_folder):
112+
return Path(local_hidden_cromshell_folder).joinpath("all.workflow.database.tsv")
113+
114+
@pytest.fixture
115+
def local_server_folder(self, local_hidden_cromshell_folder, local_cromwell_url):
116+
stripped_cromwell_url = local_cromwell_url.replace("http://", "")
117+
return Path(local_hidden_cromshell_folder).joinpath(stripped_cromwell_url)

tests/resources/README.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Testing Resources
2+
3+
### cromwell_application.conf
4+
The cromshell_application.conf file is used by cromwell during the intgratin testing.
5+
The file contains setting used by the cromwell server container to connect with an
6+
existing mysql container. A description of the config file can be found [here](https://cromwell.readthedocs.io/en/stable/tutorials/ConfigurationFiles/).

0 commit comments

Comments
 (0)