Skip to content

Commit 7907a78

Browse files
cmhulbertcmhulbert
committed
misc: script for janelia cluster
1 parent 797beb6 commit 7907a78

4 files changed

Lines changed: 181 additions & 135 deletions

File tree

‎README.md‎

Lines changed: 52 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,48 @@
44
# Paintera Conversion Helper
55
Script to assist conversion of n5 datasets to paintera-friendly formats, as specified [here](https://github.com/saalfeldlab/paintera/issues/61).
66

7-
## Installation
8-
Releases can be downloaded for Ubuntu and MacOS from the [Github Releases](https://github.com/saalfeldlab/paintera-conversion-helper/releases)
7+
## Prebuilt Releases
8+
Prebuilt releases can be downloaded for Ubuntu and MacOS from the [Github Releases](https://github.com/saalfeldlab/paintera-conversion-helper/releases)
99

10-
## Usage
10+
---
11+
## <u>Run from Source</u>
12+
### Compile
13+
To compile the conversion helper into a jar, simply run
14+
```
15+
./mvnw package
16+
```
17+
18+
To run locally:
19+
```
20+
./mvnw -q compile exec:java \
21+
-Dexec.args="to-paintera --container=in.zarr -d labels/s0 \
22+
--output-container=out.n5 --target-dataset=labels --type=label \
23+
--scale 2,2,2 2,2,2 --block-size=32,32,32"
24+
```
25+
26+
### Janelia cluster
27+
28+
Clone the repository with submodules:
29+
```
30+
git clone --recursive https://github.com/saalfeldlab/paintera-conversion-helper.git
31+
```
32+
If you have already cloned the repository, run this after cloning to fetch the submodules:
33+
```
34+
git submodule update --init --recursive
35+
```
36+
37+
For submitting a job to the Janelia cluster you can use the following script:
38+
```
39+
./startup-scripts/flintstone-paintera-convert.sh <flintstone args> -- <paintera-convert args>
40+
```
41+
Importantly:
42+
- the first flintstone arg before `--` is required, and MUST be the number of cluster nodes to use (e.g. `5`)
43+
- the classpath, main class, and additional args are provided to flintstone, so you do not need to specify them
44+
45+
46+
---
47+
## <u>Usage</u>
48+
### To Paintera
1149
This conversion tool currently supports any number of datasets (raw or label) with a
1250
single (global) block size, and will output to a single N5 group in a paintera-compatible
1351
format.
@@ -16,15 +54,8 @@ By default, spark will run locally with up to 24 workers. You can specify more w
1654
```
1755
paintera-convert to-paintera [...]
1856
```
57+
#### Example
1958

20-
`paintera-convert` can also convert paintera label sources to scalar label datasets:
21-
```
22-
paintera-convert to-scalar [...]
23-
```
24-
`to-scalar` will extract the highest resolution scale level of a Paintera dataset as a scalar `uint64` Dataset. This is useful for using Paintera painted labels (and assignments) in downstream processing, e.g. classifier training. Optionally, the `fragment-segment-assignment` can be considered and additional assignments can be added. See `to-scalar --help` for more details.
25-
26-
27-
### Usage Example
2859
To convert the `raw` and `neuron_ids` datasets of [sample A of the cremi challenge](https://cremi.org/data/) into Paintera format with mipmaps on Linux, assuming that you downloaded the data into `$HOME/Downloads`, run:
2960
```sh
3061
paintera-convert to-paintera \
@@ -38,8 +69,9 @@ paintera-convert to-paintera \
3869
--dataset-resolution 4,4,40.0 \
3970
-d volumes/labels/neuron_ids
4071
```
72+
<details>
73+
<summary><b>Usage Help </b></summary>
4174

42-
### Usage Help
4375
```
4476
$ paintera-convert to-paintera --help
4577
Usage: paintera-convert to-paintera [[--block-size=X,Y,Z|U] [--scale=X,Y,Z|U...] [--scale=X,Y,Z|U...]...
@@ -99,50 +131,17 @@ Options:
99131
--type=TYPE
100132
--help
101133
```
102-
103-
104-
<details>
105-
<summary><b>Compile</b></summary>
106-
107-
## Compile
108-
To compile the conversion helper into a jar, simply run
109-
```
110-
mvn -Denforcer.skip=true clean package
111-
```
112-
113-
To run locally build a fat jar including Spark:
114-
```
115-
mvn -Denforcer.skip=true -PfatWithSpark clean package
116-
```
117-
118-
To run on the Janelia cluster build a fat jar without Spark:
119-
```
120-
mvn -Denforcer.skip=true -Pfat clean package
121-
```
122-
123134
</details>
124135

125-
### Janelia cluster
126-
127-
Clone the repository with submodules:
128-
```
129-
git clone --recursive https://github.com/saalfeldlab/paintera-conversion-helper.git
130-
```
131-
If you have already cloned the repository, run this after cloning to fetch the submodules:
132-
```
133-
git submodule update --init --recursive
134-
```
136+
___
135137

136-
Then, run the following script to build the package:
137-
```
138-
./build-for-cluster.py
139-
```
138+
### To Scalar
140139

141-
For submitting a job to the Janelia cluster you can use the following script:
140+
`paintera-convert` can also convert from paintera label sources to scalar label datasets:
142141
```
143-
startup-scripts/spark-janelia/convert.py <number of cluster nodes> <other parameters>
142+
paintera-convert to-scalar [...]
144143
```
145-
The first parameter is the number of cluster nodes to use (for example, 5), and the rest is the same parameters as in the `paintera-convert` command.
146-
147-
148-
144+
`to-scalar` will extract the highest resolution scale level of a Paintera dataset as a multiscale `uint64` dataset.
145+
This is useful for using Paintera painted labels (and assignments) in downstream processing, e.g. classifier training.
146+
Optionally, the `fragment-segment-assignment` can be considered and additional assignments can be added.
147+
See `to-scalar --help` for more details.

‎build-for-cluster.py‎

Lines changed: 0 additions & 15 deletions
This file was deleted.

‎startup-scripts/flintstone-paintera-convert.sh‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
#!/usr/bin/env bash
2+
3+
# Launch paintera-convert on the Janelia cluster via spark-janelia/flintstone.sh.
4+
#
5+
# If the jar is not built, it will build the jar (once).
6+
#
7+
# Args before `--` go to flintstone.
8+
# Args after `--` go to paintera-convert.
9+
#
10+
# e.g.:
11+
# ./paintera-convert-janelia.sh <flintstone args> -- <paintera-convert args>
12+
#
13+
14+
set -euo pipefail
15+
16+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
17+
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
18+
19+
export SPARK_HOME="${SPARK_HOME:-/misc/local/spark-versions/spark-3.4.1}"
20+
export JAVA_HOME="${JAVA_HOME:-/misc/sc/jdks/zulu17.38.21-ca-jdk17.0.5-linux_x64}"
21+
22+
SPARK_JANELIA="${SPARK_JANELIA:-$SCRIPT_DIR/spark-janelia}"
23+
CLASS="${CLASS:-org.janelia.saalfeldlab.conversion.PainteraConvert}"
24+
25+
usage() {
26+
cat <<EOF
27+
usage: $(basename "$0") <flintstone args> -- <paintera-convert args>
28+
29+
Submits paintera-convert to the Janelia cluster via flintstone/spark-janelia.
30+
Provides classpath and Main Class to flintstone, so they should NOT
31+
be additionally specified.
32+
33+
Args before \`--' go to flintstone;
34+
Args after \`--' go to paintera-convert.
35+
36+
Example:
37+
$(basename "$0") 5 -- to-paintera --container=in.zarr -d labels/s0 \\
38+
--output-container=out.n5 --target-dataset=labels --type=label
39+
40+
help:
41+
$(basename "$0") --help flintstone help
42+
$(basename "$0") -- --help paintera-convert help
43+
$(basename "$0") -- to-paintera --help paintera-convert to-paintera help
44+
$(basename "$0") -- to-scalar --help paintera-convert to-scalar help
45+
EOF
46+
}
47+
48+
FLINTSTONE_ARGS=()
49+
PAINTERA_ARGS=()
50+
saw_separator=0
51+
for arg in "$@"; do
52+
if [[ "$saw_separator" -eq 0 && "$arg" == "--" ]]; then
53+
saw_separator=1
54+
continue
55+
fi
56+
if [[ "$saw_separator" -eq 0 ]]; then
57+
FLINTSTONE_ARGS+=("$arg")
58+
else
59+
PAINTERA_ARGS+=("$arg")
60+
fi
61+
done
62+
63+
# no `--`: all args go to flintstone
64+
if [[ "$saw_separator" -eq 0 ]]; then
65+
if [[ "$#" -eq 0 ]]; then
66+
usage 1>&2
67+
exit 1
68+
fi
69+
exec "$SPARK_JANELIA/flintstone.sh" "$@"
70+
fi
71+
72+
if [[ "${#PAINTERA_ARGS[@]}" -eq 0 ]]; then
73+
echo "error: no paintera-convert arguments after '--'" 1>&2
74+
exit 1
75+
fi
76+
77+
# --help after `--` runs paintera-convert locally, not on the cluster
78+
help_requested=0
79+
for arg in "${PAINTERA_ARGS[@]}"; do
80+
[[ "$arg" == -h || "$arg" == --help ]] && help_requested=1
81+
done
82+
83+
# an actual submission needs the flintstone worker node count
84+
if [[ "$help_requested" -eq 0 && "${#FLINTSTONE_ARGS[@]}" -eq 0 ]]; then
85+
echo "error: missing flintstone args before '--' first args (node count) is required," \
86+
"e.g. $(basename "$0") 5 -- to-paintera ..." 1>&2
87+
exit 1
88+
fi
89+
90+
# build the jar unless target/dependency is already populated
91+
LIB_DIR="${LIB_DIR:-$PROJECT_ROOT/target/dependency}"
92+
93+
find_main_jar() {
94+
ls "$LIB_DIR"/paintera-conversion-helper-*.jar 2>/dev/null \
95+
| grep -vE '\-(sources|javadoc)\.jar$' | head -1 || true
96+
}
97+
98+
MAIN_JAR="${MAIN_JAR:-$(find_main_jar)}"
99+
100+
if [[ ! -f "$MAIN_JAR" ]]; then
101+
export MAVEN_OPTS="-XX:ActiveProcessorCount=4 -XX:MaxRAMPercentage=25 ${MAVEN_OPTS:-}"
102+
"$PROJECT_ROOT/mvnw" package -DskipTests -Pspark-provided ${MAVEN_ARGS:-}
103+
MAIN_JAR="$(find_main_jar)"
104+
fi
105+
106+
if [[ ! -f "$MAIN_JAR" ]]; then
107+
echo "error: main jar not found in $LIB_DIR after build" 1>&2
108+
exit 1
109+
fi
110+
111+
JVM_ARGS="${JVM_ARGS:---add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -XX:SoftRefLRUPolicyMSPerMB=1}"
112+
113+
# help for paintera-convert runs locally
114+
if [[ "$help_requested" -eq 1 ]]; then
115+
JAVACMD="java"
116+
[[ -x "$JAVA_HOME/bin/java" ]] && JAVACMD="$JAVA_HOME/bin/java"
117+
exec "$JAVACMD" $JVM_ARGS -cp "$LIB_DIR/*" "$CLASS" "${PAINTERA_ARGS[@]}"
118+
fi
119+
120+
# append our confs so a caller-provided SUBMIT_ARGS is preserved
121+
export SUBMIT_ARGS="${SUBMIT_ARGS:-}\
122+
--conf spark.driver.extraClassPath='$LIB_DIR/*'\
123+
--conf spark.executor.extraClassPath='$LIB_DIR/*'\
124+
--conf spark.driver.extraJavaOptions='$JVM_ARGS'\
125+
--conf spark.executor.extraJavaOptions='$JVM_ARGS'"
126+
127+
exec "$SPARK_JANELIA/flintstone.sh" \
128+
"${FLINTSTONE_ARGS[@]}" "$MAIN_JAR" "$CLASS" \
129+
"${PAINTERA_ARGS[@]}"

‎startup-scripts/paintera-convert-janelia.sh‎

Lines changed: 0 additions & 67 deletions
This file was deleted.

0 commit comments

Comments
 (0)