Ziggy release 0.9.0 (2025-04-10)

Author: wohler

CITATION.cff

@@ -1,8 +1,8 @@
 cff-version: 1.2.0
 message: "If you use this software in your research, please cite it using these metadata."
 title: Ziggy
-version: v0.8.0
-date-released: "2025-01-17"
+version: v0.9.0
+date-released: "2025-04-10"
 abstract: "Ziggy, a portable, scalable infrastructure for science data processing pipelines, is the child of the Transiting Exoplanet Survey Satellite (TESS) pipeline and the grandchild of the Kepler Pipeline."
 authors:
 - family-names: Tenenbaum

RELEASE-NOTES.md

@@ -4,6 +4,50 @@
 
 These are the release notes for Ziggy. In the change log below, we'll refer to our internal Jira key for our benefit. If the item is associated with a resolved GitHub issue or pull request, we'll add a link to that. Changes that are incompatible with previous versions are marked below. While the major version is 0, we will be making breaking changes when bumping the minor number. However, once we hit version 1.0.0, incompatible changes will only be made when bumping the major number.
 
+# v0.9.0: Getting ready for 1.0.0
+
+Since this was our last chance to rename everything, we did so. The architecture has diverged over the decades, but we've made the names and architecture a lot more consistent in this release. You'll notice some of the changes in the XML elements in your pipeline definition (the Ziggy definitions are now in etc/ziggy.d); the rest are internal. In particular, rather than using "module" in a gazillion contexts, we've teased out the differences in separate names. The algorithms are now called "steps." When those steps are incorporated into a "pipeline," they are called "nodes." We no longer use module except in the context of Python modules.
+
+It should be a bit easier to add your algorithm to Ziggy as we've reduced the amount of required glue code. The prefixes for the XML pipeline configuration files such as "pd" and "pl" are no longer required and you can now organize your pipelines as you wish in one or more files. In the UI, you'll notice that the component to turn on remote execution is now a pull-down instead of a checkbox to allow for future remote environments. The new Milan nodes at the HECC are now supported.
+
+We fixed some problems interacting with PBS, running on Mac in some environments, and performing the unit tests in time zones other than Pacific. Changing ziggy.database.port now has an effect when using the HSQLDB database and the database is now stopped after running "ziggy cluster init." The log files now roll as defined in log4j2.xml. The "Process all data" and "Process new data" radio buttons in the Edit pipeline dialog work again. We also fixed some issues in the datastore:  new directories can be created, producer-consumer queries run more quickly with a large datastore, and multiple include patterns don't always produce zero files.
+
+## New Features
+
+1. Add explicit support for modules of some languages (ZIGGY-141)
+1. Improve support for compute node management (ZIGGY-164)
+1. Update libraries (ZIGGY-211)
+1. Unify pipeline definition XML readers (ZIGGY-272)
+1. Add Milan node support (ZIGGY-491)
+1. Improve Ziggy management of Python pipeline modules (ZIGGY-494)
+1. Increase flexibility in TimestampFile class (ZIGGY-525)
+1. Change stringValue field in Parameter to database type VARCHAR (ZIGGY-527)
+1. Support subclassing better in datastore classes (ZIGGY-531)
+1. Add logging of cause for failed transition (ZIGGY-539)
+1. Automate tests (simply) (ZIGGY-540)
+1. Refactor module and ui.util packages (ZIGGY-541)
+
+## Bug Fixes
+
+1. Bad QSUB return value not handled (ZIGGY-458)
+1. Parameters not sorted  (ZIGGY-485)
+1. JNA_PATH should be set somehow, or be part of instructions (ZIGGY-496)
+1. Mcc task requires MATLAB_HOME/bin on search path (ZIGGY-497)
+1. Fix file leak in LockManager (ZIGGY-524)
+1. Update pipelines doesn't set singleSubtask field correctly (ZIGGY-532)
+1. HSQLDB database server ignores database port (ZIGGY-534)
+1. Log files not rolling (ZIGGY-535)
+1. Updated remote status not always picked up upon cluster restart  (ZIGGY-536)
+1. MCR cache directories interfere with pipeline execution (ZIGGY-537)
+1. ZiggyTaskLogTest fails when properties file not set (ZIGGY-545)
+1. Iso8601Formatter unit tests fail outside of Pacific time (ZIGGY-546)
+1. EditPipelineDialog doesn't handle option to process new tasks (ZIGGY-547)
+1. Ziggy query to find consumed files runs too slowly (ZIGGY-548)
+1. cluster init doesn't stop database (ZIGGY-550)
+1. Unable to create datastore subdirectories (ZIGGY-553)
+1. ZiggyFileUtils.listFiles() that takes inclusion/exclusion patterns not working correctly (ZIGGY-556)
+1. PbsBatchManager has silent null pointer exception (ZIGGY-562)
+
 # v0.8.0: Polishing the cannonball
 
 This release consists of lots of little improvements. We added a tool to inject your pipeline version into your pipeline and updated the "ziggy" script so that you can override properties in ziggy.properties. We worked around a couple of issues that got in the way of running Ziggy on a Mac. We also fixed a bunch of annoying problems and improved the UI.

build.gradle

@@ -7,16 +7,17 @@
 // To view a dependency graph using the taskinfo plugin, run "./gradlew tiTree build"
 
 plugins {
-    id 'com.github.spotbugs' version '5.2.+'
+    id 'com.github.spotbugs' version '6.1.+'
     id 'eclipse'
     id 'jacoco'
     id 'java'
+    id 'jvm-test-suite'
     id 'maven-publish'
-    id 'org.barfuin.gradle.taskinfo' version '2.1.+'
+    id 'org.barfuin.gradle.taskinfo' version '2.2.+'
     id 'signing'
 }
 
-defaultTasks 'assemble', 'publish', 'test'
+defaultTasks ':buildSrc:publish', 'publish', 'test'
 
 ext {
     // Location of third-party sources.
@@ -43,59 +44,54 @@ dependencies {
     // Needed to compile ziggy.
     implementation files("$rootDir/buildSrc/build/libs/buildSrc-${version}.jar")
 
-    implementation 'com.github.librepdf:openpdf:1.3.+'
-    implementation 'com.github.spotbugs:spotbugs-annotations:4.7.+'
-    implementation 'com.google.guava:guava:23.6.+'
+    implementation 'com.github.librepdf:openpdf:2.0.+'
+    implementation 'com.github.spotbugs:spotbugs-annotations:4.8.+'
+    implementation 'com.google.guava:guava:33.4.+'
     implementation 'com.jgoodies:jgoodies-forms:1.9.+'
     implementation 'com.jgoodies:jgoodies-looks:2.7.+'
-    implementation 'commons-cli:commons-cli:1.5.+'
-    implementation 'commons-codec:commons-codec:1.16.+'
-    implementation 'commons-io:commons-io:2.11.+'
-    implementation 'jakarta.xml.bind:jakarta.xml.bind-api:3.0.+'
+    implementation 'commons-cli:commons-cli:1.9.+'
+    implementation 'commons-codec:commons-codec:1.18.+'
+    implementation 'commons-io:commons-io:2.18.+'
+    implementation 'jakarta.xml.bind:jakarta.xml.bind-api:4.0.+'
     implementation 'org.apache.commons:commons-collections4:4.4'
-    implementation 'org.apache.commons:commons-compress:1.25.+'
-    implementation 'org.apache.commons:commons-configuration2:2.9.+'
-    implementation 'org.apache.commons:commons-csv:1.9.+'
+    implementation 'org.apache.commons:commons-compress:1.27.+'
+    implementation 'org.apache.commons:commons-configuration2:2.11.+'
+    implementation 'org.apache.commons:commons-csv:1.13.+'
     implementation 'org.apache.commons:commons-exec:1.4.+'
-    implementation 'org.apache.commons:commons-lang3:3.12.+'
+    implementation 'org.apache.commons:commons-lang3:3.17.+'
     implementation 'org.apache.commons:commons-math3:3.6.+'
-    implementation 'org.apache.commons:commons-text:1.11.+'
-    implementation 'org.apache.logging.log4j:log4j-core:2.20.+'
-    implementation 'org.apache.logging.log4j:log4j-slf4j2-impl:2.20.+'
+    implementation 'org.apache.commons:commons-text:1.13.+'
+    implementation 'org.apache.logging.log4j:log4j-core:2.24.+'
+    implementation 'org.apache.logging.log4j:log4j-slf4j2-impl:2.24.+'
     implementation 'org.hibernate.orm:hibernate-ant:6.2.+'
     implementation 'org.hibernate:hibernate-core:6.2.+'
-    implementation 'org.javassist:javassist:3.29.2-GA'
-    implementation 'org.jfree:jfreechart:1.0.+'
-    implementation 'org.jsoup:jsoup:1.16.+'
-    implementation 'org.netbeans.api:org-netbeans-swing-outline:RELEASE121' // see note below
+    implementation 'org.javassist:javassist:3.30.2-GA'
+    implementation 'org.jfree:jfreechart:1.5.+'
+    implementation 'org.jsoup:jsoup:1.18.+'
+    implementation 'org.netbeans.api:org-netbeans-swing-outline:RELEASE121'
     implementation 'org.slf4j:slf4j-api:2.0.+'
     implementation 'org.tros:l2fprod-properties-editor:1.3.+'
 
-    // The NetBeans library started emitting the error,
-    // "No SVG loader available for ... columns.svg"
-    // at version 122 (through version 200). Hold version at 121 until
-    // this error is fixed or a workaround is discovered.
-
     // Configuration2 declares the following as optional [1]. It's not, so it's added here.
     // Occasionally, comment out this line--if the tests pass, delete it.
     // 1. https://github.com/apache/commons-configuration/blob/master/pom.xml
-    implementation 'commons-beanutils:commons-beanutils:1.9.+'
+    implementation 'commons-beanutils:commons-beanutils:1.10.+'
 
     // Libraries built outside.
-    implementation 'outside:jarhdf5:1.12.+'
+    implementation 'outside:jarhdf5:1.14.+'
     implementation 'outside:wrapper:3.5.+'
 
     // Needed to run unit tests and at runtime.
     implementation 'org.hsqldb:hsqldb:2.7.+'
 
     // Needed to compile unit tests.
     testImplementation 'junit:junit:4.13.+'
-    testImplementation 'org.hamcrest:hamcrest:2.2'
-    testImplementation 'org.mockito:mockito-core:3.12.+'
+    testImplementation 'org.hamcrest:hamcrest:3.0'
+    testImplementation 'org.mockito:mockito-core:5.15.+'
 
     // Needed at runtime.
     runtimeOnly 'org.hibernate.orm:hibernate-hikaricp:6.2.+'
-    runtimeOnly 'org.postgresql:postgresql:42.6.+'
+    runtimeOnly 'org.postgresql:postgresql:42.7.+'
 
     // Astonishingly, for some reason configuration2 doesn't work when
     // called by MATLAB, but configuration does. Ziggy doesn't use MATLAB
@@ -106,14 +102,19 @@ dependencies {
     // The following plugin reveals some bugs in our code, but until
     // the "The following classes needed for analysis were missing" bug
     // is fixed, it is commented out for daily use.
-    // spotbugsPlugins 'com.h3xstream.findsecbugs:findsecbugs-plugin:1.12.+'
+    // spotbugsPlugins 'com.h3xstream.findsecbugs:findsecbugs-plugin:1.13.+'
 
     annotationProcessor 'org.hibernate.orm:hibernate-jpamodelgen:6.2.+'
 }
 
 tasks.withType(JavaCompile) {
     options.deprecation = true
 
+    // Avoid the Eclipse "Project XXX has no explicit encoding set" warning.
+    // Until Eclipse Buildship issue #344 is fixed, remove the warning
+    // in Eclipse by using C-1 to apply a Quick Fix on the warning message.
+    options.encoding = 'UTF-8'
+
     // Suppresses "Note: Hibernate JPA 2 Static-Metamodel Generator 6.2.32.Final" output
     logging.captureStandardError LogLevel.INFO
 }
@@ -163,8 +164,8 @@ check.dependsOn javadoc
 // The SpotBugs plugin adds spotbugsMain and spotbugsTest to the check task.
 spotbugs {
     // The SMP requires that all high priority problems are addressed before testing can commence.
-    // Set reportLevel to 'low' to reveal a handful of interesting and potential bugs.
-    reportLevel = 'high'
+    // Set reportLevel to 'LOW' to reveal a handful of interesting and potential bugs.
+    reportLevel = com.github.spotbugs.snom.Confidence.valueOf('HIGH')
 }
 
 tasks.withType(com.github.spotbugs.snom.SpotBugsTask) {

buildSrc/build.gradle

@@ -8,19 +8,20 @@ plugins {
     id 'maven-publish'
 }
 
-defaultTasks 'assemble'
+defaultTasks 'publish'
 
 repositories {
     mavenCentral()
 }
 
 dependencies {
     // Needed to compile buildSrc.
-    implementation 'com.google.guava:guava:23.6.+'
-    implementation 'commons-io:commons-io:2.11.+'
+    implementation 'com.google.guava:guava:33.4.+'
+    implementation 'commons-io:commons-io:2.18.+'
     implementation 'jakarta.xml.bind:jakarta.xml.bind-api:3.0.+'
     implementation 'org.apache.commons:commons-exec:1.4.+'
-    implementation 'org.apache.commons:commons-lang3:3.12.+'
+    implementation 'org.apache.commons:commons-lang3:3.17.+'
+
     runtimeOnly 'com.sun.xml.bind:jaxb-impl:3.0.+'
 
     // Needed to compile unit tests.
@@ -32,6 +33,18 @@ dependencies {
     implementation localGroovy()
 }
 
+tasks.withType(JavaCompile) {
+    options.deprecation = true
+
+    // Avoid the Eclipse "Project XXX has no explicit encoding set" warning.
+    // Until Eclipse Buildship issue #344 is fixed, remove the warning
+    // in Eclipse by using C-1 to apply a Quick Fix on the warning message.
+    options.encoding = 'UTF-8'
+
+    // Suppresses "Note: Hibernate JPA 2 Static-Metamodel Generator 6.2.32.Final" output
+    logging.captureStandardError LogLevel.INFO
+}
+
 test {
     testLogging  {
         showStandardStreams = true
@@ -68,10 +81,10 @@ tasks.eclipse.dependsOn(cleanEclipse)
 publishing {
     publications {
         ziggyBuildSrc(MavenPublication) {
-            groupId group
+            groupId = group
             from components.java
-            artifactId "ziggy-buildSrc"
-            version version
+            artifactId = "ziggy-buildSrc"
+            version = version
 
             // Maven doesn't understand 2.17.+ notation.
             suppressPomMetadataWarningsFor('runtimeElements')

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/Mcc.java

@@ -42,6 +42,10 @@
  * <p>
  * This class puts the variable MCC_DIR into the environment and sets it to the directory of the
  * project that has invoked this task.
+ * <p>
+ * To use this class, it is necessary for the MATLAB_HOME environment variable to be set. This is
+ * then used by the {@link TessExecTask#matlabHome()} method to determine the location of the mcc
+ * executable. MATLAB_HOME must be set to the top-level MATLAB directory.
  *
  * @author Bill Wohler
  * @author Sean McCauliff
@@ -51,13 +55,23 @@ public class Mcc extends TessExecTask {
 
     private static final Logger log = LoggerFactory.getLogger(Mcc.class);
 
+    private static final String MCC_NAME = "mcc";
+
     private FileCollection controllerFiles;
     private FileCollection additionalFiles; // added with mcc -a option
     private File outputExecutable;
     private boolean singleThreaded = true;
+    private String projectDir;
+    private File buildDir;
 
     public Mcc() {
         setEnabled(isMccEnabled());
+        try {
+            projectDir = getProject().getProjectDir().getCanonicalPath();
+        } catch (IOException e) {
+            log.error("Could not obtain directory for MCC_DIR: {}", e.getMessage(), e);
+        }
+        buildDir = getProject().getLayout().getBuildDirectory().getAsFile().get();
     }
 
     @InputFiles
@@ -128,9 +142,9 @@ public Set<File> additionalFiles() {
     public void action() {
         log.info("{}.action()", this.getClass().getSimpleName());
         File matlabHome = matlabHome();
-
-        File buildBinDir = new File(getProject().getBuildDir(), "bin");
-        List<String> command = new ArrayList<>(Arrays.asList("mcc", "-v", "-m", "-N", "-d",
+        String mcc = matlabHome.toPath().resolve("bin").resolve(MCC_NAME).toString();
+        File buildBinDir = new File(buildDir, "bin");
+        List<String> command = new ArrayList<>(Arrays.asList(mcc, "-v", "-m", "-N", "-d",
             buildBinDir.toString(), "-R", "-nodisplay", "-R", "-nodesktop"));
 
         if (isSingleThreaded()) {
@@ -196,12 +210,7 @@ public void action() {
         log.info(cmd);
 
         ProcessBuilder processBuilder = new ProcessBuilder(fullCommand);
-        try {
-            processBuilder.environment()
-                .put("MCC_DIR", getProject().getProjectDir().getCanonicalPath());
-        } catch (IOException e) {
-            log.error("Could not set MCC_DIR: {}", e.getMessage(), e);
-        }
+        processBuilder.environment().put("MCC_DIR", projectDir);
         execProcess(processBuilder);
 
         Set<PosixFilePermission> neededPermissions = new HashSet<>(

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCpp.java

@@ -101,7 +101,7 @@ public class ZiggyCpp extends DefaultTask {
      */
     public ZiggyCpp() {
         Project project = getProject();
-        ziggyCppPojo.setBuildDir(project.getBuildDir());
+        ziggyCppPojo.setBuildDir(project.getLayout().getBuildDirectory().getAsFile().get());
         ziggyCppPojo.setRootDir(pipelineRootDir(project));
         if (project.hasProperty(DEFAULT_CPP_COMPILE_OPTIONS_GRADLE_PROPERTY)) {
             ziggyCppPojo.setcppCompileOptions(ZiggyCppPojo.gradlePropertyToList(

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMex.java

@@ -104,7 +104,7 @@ public class ZiggyCppMex extends DefaultTask {
      */
     public ZiggyCppMex() {
         Project project = getProject();
-        ziggyCppMexObject.setBuildDir(project.getBuildDir());
+        ziggyCppMexObject.setBuildDir(project.getLayout().getBuildDirectory().getAsFile().get());
         ziggyCppMexObject.setRootDir(ZiggyCpp.pipelineRootDir(project));
         ziggyCppMexObject.setProjectDir(project.getProjectDir());
         if (project.hasProperty(DEFAULT_COMPILE_OPTIONS_GRADLE_PROPERTY)) {

buildSrc/src/main/java/gov/nasa/ziggy/buildutil/ZiggyCppMexPojo.java

@@ -198,10 +198,10 @@ public String generateLinkCommand() {
     /**
      * Generates a name for the shared object library in the event that none has been set. This is
      * done by taking the project name and adding to it the components of the C++ path name,
-     * separated by hyphens. For example, if the project directory is /path/to/pipeline/module1, and
-     * the source directory is /path/to/pipeline/module1/src/main/cpp/mex, the name of the library
-     * will be module1-src-main-cpp-mex, resulting in a shared object named
-     * libmodule1-src-main-cpp-mex.so or .dylib.
+     * separated by hyphens. For example, if the project directory is /path/to/pipeline/step1, and
+     * the source directory is /path/to/pipeline/step1/src/main/cpp/mex, the name of the library
+     * will be step1-src-main-cpp-mex, resulting in a shared object named
+     * libstep1-src-main-cpp-mex.so or .dylib.
      *
      * @return Generated name for the shared object library.
      */