Skip to content

Commit b1d5b6a

Browse files
bshifawbshifaw
and
bshifaw
authored
Release edits for cromshell2 (#255)
* Added short option for --cromwell-url * changed cromshell2 initiation from `cromshell-beta` to `cromshell`, minor edits to README.md * Added test badges * removed "Notify" command description for README.md * rebased cromshell sh script using master branch, moved cromshell sh script to legacy_cromshell folder * moved completions folder to legacy_cromshell folder --------- Co-authored-by: bshifaw <[email protected]>
1 parent abd12ba commit b1d5b6a

File tree

8 files changed

+412
-54
lines changed

8 files changed

+412
-54
lines changed

LICENSE

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
BSD 3-Clause License
22

3-
Copyright (c) 2018, Broad Institute
3+
Copyright (c) 2023, Broad Institute
44
All rights reserved.
55

66
Redistribution and use in source and binary forms, with or without

README.md

+31-37
Original file line numberDiff line numberDiff line change
@@ -7,36 +7,32 @@
77
"" "" "" "" "" ""
88
```
99

10-
# cromshell
11-
A CLI for submitting workflows to a cromwell server and monitoring / querying their results.
10+
# Cromshell
11+
[![GitHub version](https://badge.fury.io/gh/broadinstitute%2Fcromshell.svg)](https://badge.fury.io/gh/broadinstitute%2Fcromshell)
12+
[![Integration Test Workflow](https://github.com/broadinstitute/cromshell/actions/workflows/integration_tests.yml/badge.svg)](https://github.com/broadinstitute/cromshell/actions/workflows/integration_tests.yml/badge.svg)
13+
[![Unit Test Workflow](https://github.com/broadinstitute/cromshell/actions/workflows/unit_tests.yml/badge.svg)](https://github.com/broadinstitute/cromshell/actions/workflows/unit_tests.yml/badge.svg)
14+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
1215

13-
Current version: 2.0.0.beta
14-
15-
Cromshell 2 is the next step in the evolution Cromshell. It offers many of the same
16-
functions as Cromshell 1 but has been rebuilt in python with many added benefits such as:
17-
- Automatically zip dependencies when submitting a workflow.
18-
- Added integration and unit tests to insure reliability
19-
- Multiple ways of installation (source, brew tap, and pypi).
20-
- Modular architecture making adding new functionalities easier.
21-
- Developer documentation showing how to add new features and commands to the tool.
16+
Cromshell is a CLI for submitting workflows to a Cromwell server and monitoring/querying their results.
2217

2318
## Examples:
2419

2520
```
26-
cromshell-beta submit workflow.wdl inputs.json options.json dependencies.zip
27-
cromshell-beta status
28-
cromshell-beta -t 20 metadata
29-
cromshell-beta logs -2
21+
cromshell submit workflow.wdl inputs.json options.json dependencies.zip
22+
cromshell status
23+
cromshell -t 20 metadata
24+
cromshell logs -2
3025
```
3126

3227
## Supported Options:
3328
* `--no_turtle` or `--I_hate_turtles`
3429
* Hide turtle logo
3530
* `--cromwell_url [TEXT]`
36-
* Specify Cromwell URL used.
31+
* Specifies Cromwell URL used.
3732
* `TEXT` Example: `http://65.61.654.8:8000`
33+
* Note: Depending on your Cromwell server configuration, you may not need to specify the port.
3834
* `-t [TIMEOUT]`
39-
* Specify the server connection timeout in seconds.
35+
* Specifies the server connection timeout in seconds.
4036
* Default is 5 sec.
4137
* `TIMEOUT` must be a positive integer.
4238
* `--gcloud_token_email [TEXT]`
@@ -51,16 +47,16 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
5147

5248
#### Start/Stop workflows
5349
* `submit [-w] <wdl> <inputs_json> [options_json] [included_wdl_zip_file]`
54-
* Will automatically validate the WDL and JSON file.
55-
* Submit a new workflow.
50+
* Automatically validates the WDL and JSON file.
51+
* Submit a new workflow to the Cromwell server.
5652
* *`-w`* [COMING SOON] Wait for workflow to transition from 'Submitted' to some other status before ${SCRIPTNAME} exits.
5753
* *`included_wdl_zip_file`* Zip file containing any WDL files included in the input WDL
5854
* `abort [workflow-id] [[workflow-id]...]`
5955
* Abort a running workflow.
6056
#### Workflow information:
6157
* `alias <workflow-id> <alias_name>`
6258
* Label the given workflow ID with the given alias_name. Aliases can be used in place of workflow IDs to reference jobs.
63-
* Remove alias by passing empty double quotes as `alias_name` (e.g. `alias <workflow-id> ""`)
59+
* Remove an alias by passing empty double quotes as `alias_name` (e.g. `alias <workflow-id> ""`)
6460
#### Query workflow status:
6561
* `status [workflow-id] [[workflow-id]...]`
6662
* Check the status of a workflow.
@@ -72,8 +68,7 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
7268
* Get the summarized status of all jobs in the workflow.
7369
* `-j` prints a JSON instead of a pretty summary of the execution status (compresses subworkflows)
7470
* `-x` compress sub-workflows for less detailed summarization
75-
* `timing` *`[workflow-id] [[workflow-id]...]`*
76-
71+
* `timing` *`[workflow-id] [[workflow-id]...]`*
7772
* Open the timing diagram in a browser.
7873

7974
#### Logs
@@ -83,26 +78,22 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
8378
* Download all logs produced by a workflow.
8479

8580
#### Job Outputs
86-
* [COMING SOON] `list-outputs [workflow-id] [[workflow-id]...]`
81+
* `list-outputs [workflow-id] [[workflow-id]...]`
8782
* List all output files produced by a workflow.
8883
* [COMING SOON] `fetch-all [workflow-id] [[workflow-id]...]`
8984
* Download all output files produced by a workflow.
90-
91-
#### Get email notification on job completion
92-
* [COMING SOON] `notify [workflow-id] [daemon-server] email [cromwell-server]`
93-
* *`daemon-server`* server to run the notification daemon on
9485

9586
#### Display a list jobs submitted through cromshell
9687
* `list [-c] [-u]`
97-
* `-c` Color the output by completion status
98-
* `-u` Check completion status of all unfinished jobs
88+
* `-c` Color the output by completion status.
89+
* `-u` Check completion status of all unfinished jobs.
9990

10091
#### Clean up local cached list
10192
* [COMING SOON] `cleanup [-s STATUS]`
10293
* Remove completed jobs from local list.
103-
Will remove all jobs from the local list that are in a completed state,
94+
This command removes all jobs from the local list that are in a completed state,
10495
where a completed state is one of: `Succeeded`, `Failed`, `Aborted`
105-
* *`-s STATUS`* If provided, will only remove jobs with the given `STATUS` from the local list.
96+
* *`-s [STATUS]`* If provided, will only remove jobs with the given `[STATUS]` from the local list.
10697

10798
#### Update cromwell server
10899
* `update-server`
@@ -127,9 +118,9 @@ functions as Cromshell 1 but has been rebuilt in python with many added benefits
127118
You may omit the job ID of the last job submitted when running commands, or use negative numbers to reference previous jobs, e.g. "-1" will track the last job, "-2" will track the one before that, and so on.
128119
* You can override the default cromwell server by setting the argument `--cromwell_url` to the appropriate URL.
129120
* You can override the default cromshell configuration folder by setting the environmental variable `CROMSHELL_CONFIG` to the appropriate directory.
130-
* Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `./cromshell status -- -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`).
131-
* Assign aliases to workflow ids using the alias command (i.e. `./cromshell alias -- -1 myAliasName`).
132-
* Once the Alias command is used to attach an alias to a workflow id, the alias name can be used instead of the id (i.e. `./cromshell status myAliasName`).
121+
* Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `cromshell status -- -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`).
122+
* Assign aliases to workflow ids using the alias command (i.e. `cromshell alias -- -1 myAliasName`).
123+
Once the Alias command is used to attach an alias to a workflow id, the alias name can be used instead of the id (i.e. `cromshell status myAliasName`).
133124

134125
## Installation
135126
From brew
@@ -145,10 +136,9 @@ From source
145136

146137
git clone [email protected]:broadinstitute/cromshell.git
147138
cd cromshell
148-
git checkout cromshell_2.0
149139
pip install .
150140

151-
cromshell-beta --help
141+
cromshell --help
152142

153143
## Uninstallation
154144
From brew
@@ -157,10 +147,14 @@ From brew
157147

158148
From pypi/source
159149

160-
pip uninstall cromshell-beta
150+
pip uninstall cromshell
161151

162152
## Development
163153

164154
See the [Developer Docs](./developer_docs/)
165155

156+
## Legacy Cromshell
166157

158+
The original Cromshell shell script is still available in the legacy_cromshell folder and in the `cromshell1` branch of this repository.
159+
It is no longer maintained, but is still available for use. The original Cromshell contains some commands not yet available in Cromshell2,
160+
such as `fetch-logs`, `fetch-all`, `notify`, and `cleanup`. These commands will be added to Cromshell2 in the future.

legacy_cromshell/README.md

+130
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Legacy Cromshell
2+
3+
The legacy cromshell folder contains the original cromshell script. It is no longer maintained and is only kept for historical purposes.
4+
5+
---
6+
7+
```
8+
__ __
9+
.,-;-;-,. /'_\ +-----------------------------------------------+ /_'\.,-;-;-,.
10+
_/_/_/_|_\_\) / | CROMSHELL : run Cromwell jobs from the shell | \ (/_/__|_\_\_
11+
'-<_><_><_><_>=/\ +-----------------------------------------------+ /\=<_><_><_><_>-'
12+
`/_/====/_/-'\_\ /_/'-\_\====\_\'
13+
"" "" "" "" "" ""
14+
```
15+
16+
# Installation
17+
18+
`cromshell` and its dependencies can be installed on OSX with `brew install broadinstitute/dsp/cromshell`
19+
20+
or through [bioconda](https://bioconda.github.io/) with `conda install cromshell`
21+
22+
Alternatively, download the script and put it somewhere...
23+
24+
# cromshell
25+
A script for submitting workflows to a cromwell server and monitoring / querying their results.
26+
27+
requires `column`, `curl`, `mail`, and [jq](https://stedolan.github.io/jq/)
28+
29+
### Examples:
30+
31+
```
32+
cromshell submit workflow.wdl inputs.json options.json dependencies.zip
33+
cromshell status
34+
cromshell -t 20 metadata
35+
cromshell logs -2
36+
```
37+
38+
### Supported Flags:
39+
* `-t` `TIMEOUT`
40+
* Set the curl connect timeout to `TIMEOUT` seconds.
41+
* Also sets the curl max timeout to `2*TIMEOUT` seconds.
42+
* `TIMEOUT` must be an integer.
43+
44+
### Supported Subcommands:
45+
46+
47+
#### Start/Stop workflows
48+
* `submit` `[-w]` *`<wdl>`* *`<inputs_json>`* `[options_json]` `[included_wdl_zip_file]`
49+
* Submit a new workflow.
50+
* Will automatically validate the WDL and JSON file if `womtool` is in your path.
51+
* To add `womtool` to your path, install `cromwell` with brew:
52+
* `brew install cromwell`
53+
* *`-w`* Wait for workflow to transition from 'Submitted' to some other status
54+
before ${SCRIPTNAME} exits
55+
* *`included_wdl_zip_file`* Zip file containing any WDL files included in the input WDL
56+
* `abort` *`[workflow-id] [[workflow-id]...]`*
57+
* Abort a running workflow.
58+
#### Workflow information:
59+
* `alias` *`<workflow-id>` `<alias_name>`*
60+
* Label the given workflow ID with the given alias_name. Aliases can be used in place of workflow IDs to reference jobs.
61+
#### Query workflow status:
62+
* `status` *`[workflow-id] [[workflow-id]...]`*
63+
* Check the status of a workflow.
64+
* `metadata` *`[workflow-id] [[workflow-id]...]`*
65+
* Get the full metadata of a workflow.
66+
* `slim-metadata` *`[workflow-id] [[workflow-id]...]`*
67+
* Get a subset of the metadata from a workflow.
68+
* `execution-status-count`, `counts` *`[-p] [-x] [workflow-id] [[workflow-id]...]`*
69+
* Get the summarized status of all jobs in the workflow.
70+
* `-p` prints a pretty summary of the execution status instead of JSON
71+
* `-x` expands sub-workflows for more detailed summarization
72+
* `timing` *`[workflow-id] [[workflow-id]...]`*
73+
* Open the timing diagram in a browser.
74+
75+
#### Logs
76+
* `logs` *`[workflow-id] [[workflow-id]...]`*
77+
* List the log files produced by a workflow.
78+
* `fetch-logs` *`[workflow-id] [[workflow-id]...]`*
79+
* Download all logs produced by a workflow.
80+
81+
#### Job Outputs
82+
* `list-outputs` *`[workflow-id] [[workflow-id]...]`*
83+
* List all output files produced by a workflow.
84+
* `fetch-all` *`[workflow-id] [[workflow-id]...]`*
85+
* Download all output files produced by a workflow.
86+
87+
#### Get email notification on job completion
88+
* `notify` *`[workflow-id]` `[daemon-server]` `email` `[cromwell-server]`*
89+
* *`daemon-server`* server to run the notification daemon on
90+
91+
#### Display a list jobs submitted through cromshell
92+
* `list` *`[-c]` `[-u]`*
93+
* *`-c`* Color the output by completion status
94+
* *`-u`* Check completion status of all unfinished jobs
95+
96+
#### Clean up local cached list
97+
* `cleanup` *`[-s STATUS]`*
98+
* Remove completed jobs from local list.
99+
Will remove all jobs from the local list that are in a completed state,
100+
where a completed state is one of: `Succeeded`, `Failed`, `Aborted`
101+
* *`-s STATUS`* If provided, will only remove jobs with the given `STATUS` from the local list.
102+
103+
#### Update cromwell server
104+
* `update-server`
105+
* Change the cromwell server that new jobs will be submitted to.
106+
107+
#### Get cost of a workflow
108+
Costs are only available for workflows that completed more than 8 hours ago on a `GCS` backend.
109+
Requires the `~/.cromshell/gcp_bq_cost_table.config` configuration file to exist and contain the name of the BigQuery cost table for your organization.
110+
Also ensure that the default project has been set using `bq init`.
111+
* `cost [workflow-id] [[workflow-id]...]`
112+
* Get the cost for a workflow.
113+
* `cost-detailed [workflow-id] [[workflow-id]...]`
114+
* Get the cost for a workflow at the task level.
115+
116+
117+
### Features:
118+
* Running `submit` will create a new folder in the `~/.cromshell/${CROMWELL_URL}/` directory named with the cromwell job id of the newly submitted job.
119+
It will copy your wdl and json inputs into the folder for reproducibility.
120+
* It keeps track of your most recently submitted jobs by storing their ids in `./cromshell/`
121+
You may omit the job ID of the last job submitted when running commands, or use negative numbers to reference previous jobs, e.g. "-1" will track the last job, "-2" will track the one before that, and so on.
122+
* You can override the default cromwell server by setting the environmental variable `CROMWELL_URL` to the appropriate URL.
123+
* Most commands takes multiple workflow-ids, which you *can specify both in relative and absolute ID value* (i.e. `./cromwell status -1 -2 -3 c2db2989-2e09-4f2c-8a7f-c3733ae5ba7b`).
124+
* You can supply additional headers to Cromshell REST calls by setting the environmental variable `CROMSHELL_HEADER`. This is useful if your Cromwell server is fronted by an auth server that authenticates access using bearer tokens before forwarding requests onto the Cromwell API. For example: `CROMSHELL_HEADER="Authorization: Bearer 3e2f34f2e..."`
125+
126+
### Code Conventions:
127+
Please try to follow these conventions when editing cromshell.
128+
* Use double brackets for tests ( `[[ ... ]]` instead of `[]`)
129+
* Use `{}` when doing dereferencing variables (`${VALUE}`,`${1}` instead of `$VALUE`,`$1`)
130+
* Define functions with the `function` keyword (`function doThing()` instead of `doThing()`)

completions/_cromshell renamed to legacy_cromshell/completions/_cromshell

+4-1
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ function _cromshell {
2323
notify\:'Get email notification on job completion'
2424
list\:'Display a list jobs submitted through cromshell'
2525
cleanup\:'Clean up local cached list'
26+
cost\:'Display workflow cost'
27+
cost-detailed\:'Display workflow cost broken down by task'
28+
update-server\:'Change the cromshell server new jobs will be submited to'
2629
))" \
2730
"*::arg:->args"
2831

@@ -33,7 +36,7 @@ function _cromshell {
3336
alias)
3437
_alias
3538
;;
36-
abort|status|metadata|slim-metadata|timing|logs|fetch-logs|list-outputs|fetch-all)
39+
abort|status|metadata|slim-metadata|timing|logs|fetch-logs|list-outputs|fetch-all|cost|cost-detailed)
3740
_id_list_function
3841
;;
3942
list)
+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Reload the _cromshell completion file for testing
2+
# Usage:
3+
# source reload.sh
4+
5+
fpath=($PWD $fpath)
6+
unfunction _cromshell
7+
autoload -U compinit
8+
compinit

0 commit comments

Comments
 (0)