Skip to content

Commit a4e6c7f

Browse files
author
Bruno Grande
authored
Merge pull request #6 from Sage-Bionetworks-Workflows/bgrande/ORCA-90/final-touches
[ORCA-90] Implement missing functionality for a user acceptance test
2 parents bbe0855 + 8267bfd commit a4e6c7f

File tree

19 files changed

+1519
-181
lines changed

19 files changed

+1519
-181
lines changed

‎.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
__pycache__/*
1212
.cache/*
1313
.*.swp
14-
*/.ipynb_checkpoints/*
14+
.ipynb_checkpoints
1515
.DS_Store
1616
/tests/outputs/
1717
test.txt

‎.pre-commit-config.yaml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,10 @@ repos:
1717
- id: mixed-line-ending
1818
args: ['--fix=auto'] # replace 'auto' with 'lf' to enforce Linux/Mac line endings or 'crlf' for Windows
1919

20-
- repo: https://github.com/myint/autoflake
21-
rev: v1.4
20+
- repo: https://github.com/PyCQA/autoflake
21+
rev: v2.0.0
2222
hooks:
2323
- id: autoflake
24-
args: [
25-
--in-place,
26-
--remove-all-unused-imports,
27-
--remove-unused-variables,
28-
]
2924

3025
- repo: https://github.com/pycqa/isort
3126
rev: 5.11.4
@@ -54,3 +49,8 @@ repos:
5449
rev: 'v0.991'
5550
hooks:
5651
- id: mypy
52+
53+
- repo: https://github.com/kynan/nbstripout
54+
rev: '0.6.1'
55+
hooks:
56+
- id: nbstripout

‎Pipfile.lock‎

Lines changed: 710 additions & 94 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@
22

33
<!--
44
[![ReadTheDocs](https://readthedocs.org/projects/dcqc/badge/?version=latest)](https://sage-bionetworks-workflows.github.io/dcqc/)
5-
[![PyPI-Server](https://img.shields.io/pypi/v/dcqc.svg)](https://pypi.org/project/dcqc/)
65
-->
6+
[![PyPI-Server](https://img.shields.io/pypi/v/dcqc.svg)](https://pypi.org/project/dcqc/)
7+
[![codecov](https://codecov.io/gh/Sage-Bionetworks-Workflows/py-dcqc/branch/main/graph/badge.svg?token=OCC4MOUG5P)](https://codecov.io/gh/Sage-Bionetworks-Workflows/py-dcqc)
78
[![Project generated with PyScaffold](https://img.shields.io/badge/-PyScaffold-005CA0?logo=pyscaffold)](#pyscaffold)
89

910
> Python package for performing quality control (QC) for data coordination (DC)

‎demos/DCQC.ipynb‎

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "1af2c847-7f27-4365-bc1b-80c9f34bb190",
6+
"metadata": {},
7+
"source": [
8+
"## Import modules"
9+
]
10+
},
11+
{
12+
"cell_type": "code",
13+
"execution_count": null,
14+
"id": "eee68e50-439a-4aea-b2d6-741492fb9f15",
15+
"metadata": {},
16+
"outputs": [],
17+
"source": [
18+
"import json\n",
19+
"from pathlib import Path\n",
20+
"\n",
21+
"import pytest\n",
22+
"\n",
23+
"from dcqc.parsers import CsvParser\n",
24+
"from dcqc.reports import JsonReport\n",
25+
"from dcqc.tests.test_abc import TestABC\n",
26+
"from dcqc.utils import open_parent_fs"
27+
]
28+
},
29+
{
30+
"cell_type": "markdown",
31+
"id": "c737582c-d0c3-4087-b848-9ead1404a702",
32+
"metadata": {},
33+
"source": [
34+
"## GIVEN a list of external tests to skip (to remain self-contained)"
35+
]
36+
},
37+
{
38+
"cell_type": "code",
39+
"execution_count": null,
40+
"id": "3a9cad4a-ddd4-4096-adbe-40e779aac644",
41+
"metadata": {},
42+
"outputs": [],
43+
"source": [
44+
"all_tests = TestABC.list_tests()\n",
45+
"skipped_tests = [test.__name__ for test in all_tests if test.is_external_test]"
46+
]
47+
},
48+
{
49+
"cell_type": "markdown",
50+
"id": "367994d3-eb4a-4b5e-8935-65f2e2266cd3",
51+
"metadata": {},
52+
"source": [
53+
"## AND a subset of internal tests to be required (to verify suite status behavior)"
54+
]
55+
},
56+
{
57+
"cell_type": "code",
58+
"execution_count": null,
59+
"id": "641ebc10-d3bf-486c-9315-d46c811eb711",
60+
"metadata": {},
61+
"outputs": [],
62+
"source": [
63+
"required_tests = [\"Md5ChecksumTest\"]"
64+
]
65+
},
66+
{
67+
"cell_type": "markdown",
68+
"id": "65dda6a2-ee0f-4384-a7d5-202332ead35c",
69+
"metadata": {},
70+
"source": [
71+
"## AND a CSV file of TXT and TIFF files"
72+
]
73+
},
74+
{
75+
"cell_type": "code",
76+
"execution_count": null,
77+
"id": "a5de4997-c9ac-4084-b435-33fbbad5bf80",
78+
"metadata": {},
79+
"outputs": [],
80+
"source": [
81+
"csv_path = Path(\"../tests/data/files.csv\")"
82+
]
83+
},
84+
{
85+
"cell_type": "markdown",
86+
"id": "1e9755cc-54a0-4e7d-ab2b-14b981d4e7c4",
87+
"metadata": {},
88+
"source": [
89+
"## AND a remote destination for the JSON report"
90+
]
91+
},
92+
{
93+
"cell_type": "code",
94+
"execution_count": null,
95+
"id": "7321a4a8-a6cf-44a6-b4d9-f72fa7f52e1c",
96+
"metadata": {},
97+
"outputs": [],
98+
"source": [
99+
"report_url = \"syn://syn50696607/report.json\""
100+
]
101+
},
102+
{
103+
"cell_type": "markdown",
104+
"id": "7cb3340e-50e2-4798-a17c-2dddaf4445b3",
105+
"metadata": {},
106+
"source": [
107+
"## WHEN the CSV file is parsed to generate the relevant QC suites"
108+
]
109+
},
110+
{
111+
"cell_type": "code",
112+
"execution_count": null,
113+
"id": "05d29389-794b-4895-a809-8fb557ff9568",
114+
"metadata": {},
115+
"outputs": [],
116+
"source": [
117+
"parser = CsvParser(csv_path)\n",
118+
"suites = parser.create_suites(required_tests, skipped_tests)"
119+
]
120+
},
121+
{
122+
"cell_type": "markdown",
123+
"id": "9773bf52-f224-4313-bcf0-9c81c2aa6143",
124+
"metadata": {},
125+
"source": [
126+
"## AND those suites are used to generate a JSON report"
127+
]
128+
},
129+
{
130+
"cell_type": "code",
131+
"execution_count": null,
132+
"id": "57f77b89-e058-4c4f-b7af-6a7e88a4950a",
133+
"metadata": {},
134+
"outputs": [],
135+
"source": [
136+
"report = JsonReport(report_url, overwrite=True)\n",
137+
"saved = report.save(suites)"
138+
]
139+
},
140+
{
141+
"cell_type": "markdown",
142+
"id": "e38cc15c-4992-4038-abb7-26339d7c1da8",
143+
"metadata": {},
144+
"source": [
145+
"## THEN the file exists"
146+
]
147+
},
148+
{
149+
"cell_type": "code",
150+
"execution_count": null,
151+
"id": "87ac0c7d-36c0-4a2f-b3b5-480f2e8c31cc",
152+
"metadata": {},
153+
"outputs": [],
154+
"source": [
155+
"fs, basename = open_parent_fs(report_url)\n",
156+
"assert fs.exists(basename)"
157+
]
158+
},
159+
{
160+
"cell_type": "markdown",
161+
"id": "334c8073-7812-43fb-be57-3354f6f3c489",
162+
"metadata": {},
163+
"source": [
164+
"## AND the file can be loaded by the `json` module"
165+
]
166+
},
167+
{
168+
"cell_type": "code",
169+
"execution_count": null,
170+
"id": "40d2dacf-f39e-4433-9495-d730567b12d1",
171+
"metadata": {},
172+
"outputs": [],
173+
"source": [
174+
"with fs.open(basename) as infile:\n",
175+
" contents = json.load(infile)\n",
176+
"assert contents"
177+
]
178+
},
179+
{
180+
"cell_type": "markdown",
181+
"id": "de5a4cff-994d-4608-86c1-16f5da61bfc2",
182+
"metadata": {},
183+
"source": [
184+
"## Print the report"
185+
]
186+
},
187+
{
188+
"cell_type": "code",
189+
"execution_count": null,
190+
"id": "b646e29d-dc31-49b8-a627-84aaab661d4e",
191+
"metadata": {
192+
"tags": []
193+
},
194+
"outputs": [],
195+
"source": [
196+
"contents_fmt = json.dumps(contents)\n",
197+
"\n",
198+
"print(contents_fmt)"
199+
]
200+
},
201+
{
202+
"cell_type": "code",
203+
"execution_count": null,
204+
"id": "1b4e26c5-c12c-4d89-bd19-dc43549c989b",
205+
"metadata": {},
206+
"outputs": [],
207+
"source": []
208+
}
209+
],
210+
"metadata": {
211+
"kernelspec": {
212+
"display_name": ".venv",
213+
"language": "python",
214+
"name": "python3"
215+
},
216+
"language_info": {
217+
"codemirror_mode": {
218+
"name": "ipython",
219+
"version": 3
220+
},
221+
"file_extension": ".py",
222+
"mimetype": "text/x-python",
223+
"name": "python",
224+
"nbconvert_exporter": "python",
225+
"pygments_lexer": "ipython3",
226+
"version": "3.10.4 (main, Jul 7 2022, 15:39:05) [Clang 12.0.5 (clang-1205.0.22.11)]"
227+
},
228+
"vscode": {
229+
"interpreter": {
230+
"hash": "b3cc5ad3e41ecdab922a1c2f15e033ac62b3a1f322e57b79d2f2f637461ca57a"
231+
}
232+
}
233+
},
234+
"nbformat": 4,
235+
"nbformat_minor": 5
236+
}

0 commit comments

Comments
 (0)