Enables more tests, including server level tests with Jetty (#154)

* Add WEB-INF/temp to gitignore

* Set up a way to run tests dependent on WEB-INF directory structure

* Migrate SgtMap tests to JUnit

Note many of the images showed small differences from old tests due to fonts. I added a new tag that's just informative that these tests are font dependent (the images I'm adding pass on my machine-  there might be issues on different machines).

* Migrate some more tests to JUnit

New tag RequiresContent for tests that are relying on the content/setup.xml, I still need to find a way to make those tests not require manual setup.

* migrate more tests, add support for skipping setup.xml validation

Skipping the setup.xml validation allows additional tests that don't rely on those values

* Remove lastmodified column from the file visitor tests

Last modified is difficult to have consistent across machines (and even on the same machine).

* New load datasets using xml fragments approach and helper class for that.

Also migrate EDDGridFromDap to JUnit.

Sadly most tests are reliant on datasets from THREDDS, so they still aren't running. The plan to fix that is one of: mock THREDDS responses, local files for those datasets, or different datasets.

* Add support for generating test code coverage (based on JUnit tests)

* Migrate tests for cohort/array, cohort/util

* Migrate TestUtil and DataStream to JUnit

* Migrate a large collection of tests to junit

* Move resources to not included in git

Planning to transition to downloading the test resource files.

* Migrate more code to junit, update test resources for a plan to handle large files

Also moves some files to a new scripts folder that is for code designed to be run manually that is not part of the main server.

* Another set of tests migrated to junit

* More test migrations

Also update image all comparisons to use resource paths.

* Split up test resource files, Mark Flaky tests, split off very large files

This is to make the folders small enough they can be zipped under 2gb and served from a git release

* New approach for maven downloading files (using the maven-download-plugin now)

Also adding flaky annotations for tests that failed (several of them I believe failed due to things like file last modified timestamp being different).

* Update EDDGridFromDapTests.java

* Change when test resources are downloaded

Add the flaky annotation to several more tests.

* Improve netcdf loading for a fresh install, one more flaky tag

* More changes to eliminate reliance on specific hardcoded paths

Also fix a couple hardcoded dataset ids to use suggested ids.

* Centralize code for test initialization. Also set tests to use SansSerif for tests.

Tests are now using SansSerif because it is always available in java distributions and so it eliminates one more step needed for developer setup.

* For setups without fonts installed, any use of EDStatic also needs the font setup

* Add 2024 to the FileVisitorDNLS gpcp test

* Re-add download of content

* Set file separators to forward slash

* Update FileVisitorSubdirTests for path slash changes

* Update warsourceExcludes for new folders

* fix HashDigestTests path

* Fix some jUnit tests on Linux (#7)

* Fix some jUnit tests on Linux

Fix many path related and other issues with jUnit tests on Linux.

Note that these changes don't yet make every test pass, there
are still issues with font differences in image generation,
non-portable performance checks, and a handful of other differences.

* Fix watch directory tests on windows

While linux may not send the modify events on deletion or the directory event, windows does

* Disable the display in browser, it causes tests to pause

* Have the path regex work for both windows and linux

* Fix the forwardSlashDir utility function

This is only used in one spot, so should be safe to modify.
Also it was previously checking if the string started with a / and if not, adding a slash to the end of it.

* Fix some of the EDDGridFromNcFilesTests on Linux

The wording of FileNotFoundException differs, so just make sure its that kind of exception. Mark testNcml flaky (for now), and fix the capitilzation of testGridNThreads for systems that are sensitive to that.

* Disable the performance part of persistentTableTests for now

* Change to using a temp dir for eddtablefromhttpgettests

---------

Co-authored-by: Chris John <chris.john@noaa.gov>

* Clean up no longer used unitTestDataDir (and big version)

* Add a tag for imageComparison tests.

Add some explanation for image comparison tests to the readme.

* Mark a few tests as flaky due to differences on different OS. Also remove the performance checks in a couple spots.

* Don't start the email thread if we're testing

* Fix the regexfilename tests that look for java files

Recent pom changes mean the java files aren't copied over to the build outputs

* Fix the EDDTests to use the values in the Jetty setup.xml

* Simplify the changes when running in a test environment

* Re-add the ability for tests to skip defered loading

* Remove Flaky tag from several file visitor tests

* Have directory_strlen work cross platform

* Because file selection for attributes is based on OS lastModified and there's discepencies on how OSes handle that field, different files can be selected for the attributes of a dataset. Update some tests that were running into this with pulling some attributes from the loaded data instead of fully hardcoded.

* Fix a handful of flaky tests

The main changes are to remove comparisons on last modified time
There's a couple other small changes as well (sorting results in a few tests to make sure the lines are always in the same order for example)

* Remove TagFlaky from tests that are fixed or currently seem to be working

Some of them were external servers that had previously temporarily been unavailable.

* Fix attributes for test1d

* First version of running a test that requires local erddap using jetty

* Enable more local erddap tests using jetty

* Generate a datasets file to use with jetty in tests that contains all of the test datasets

* Migrate additional tests to Jetty

Fix some flaky tests

* Make some tests more resilient to minor changes in data (use regex to not test some numbers).

* Enable some additional tests (SOS, Thredds, wms)

* Add initialization to EDDTableFromFilesTests

* Enable more tests

Get several local ERDDAP/Jetty tests working.
Change some tests from discontinued thredds datasets to new datasets and turn them on.

* New tests to cover gaps (metadata, sitemap, interpolate)

* Format EDDTestDataset

* Add test to make sure aggregaterows uses range from all inputs

* Test for esriAscii also format EDDGridTests and EDDTableAggregateRowsTests

* Enabling additional tests

As well as decrease flakiness of some existing tests.

* Add support for mismatched slashses to fileVisitor reduceDnlsTableToOneDir

* Add support for file separator mismatches while using files functionality

Enable an additional Jetty Test

* More changes to make tests more stable and pass cross platform

* More test changes to reduce flakiness

* Migrating some of the code not used by ERDDAP into separate directories

Possibly should delete this, but want to keep it until that's confirmed

* Remove jdom from repo

All tests still pass, should get additional confirmation on geotiff writer before release

* Archive additional code

Mostly dods/servers, dods/servlets. Also some additional files that were used with Browsers.

* Remove several datasets that fail to load from the Jetty tests datasets file

Also reduce the load wait time to 10 minutes.

* Fix filename filter test after code was archived

* Better testing geotiff

---------

Co-authored-by: Shane St Savage <shane@axds.co>
Co-authored-by: Shane St Savage <shane@axiomdatascience.com>

Author: ChrisJohnNOAA

.gitignore

@@ -17,3 +17,6 @@ Thumbs.db
 .project
 src/test/resources/
 .idea
+development/test/datasets.xml
+development/test/datasets.xml*
+missingImage.txt

WEB-INF/classes/gov/noaa/pfel/coastwatch/TestAll.java

@@ -5,7 +5,6 @@
 package gov.noaa.pfel.coastwatch;
 
 import com.cohort.array.*;
-import com.cohort.ema.*;
 import com.cohort.util.*;
 
 import gov.noaa.pfel.coastwatch.*;
@@ -827,21 +826,11 @@ public static void main(String args[]) throws Throwable {
 Attributes att;       
 AttributedString2 as2;
 Boundaries boun;
-Browser browser;
 ByteArray ba;
 Calendar2 calendar2; 
 CharArray chara;
 CompoundColorMap ccm;
 CompoundColorMapLayerChild ccmlc;
-ContourScreen cons;
-CWUser cwUser;
-CWBrowser cwBrowser;
-//CWBrowserHAB cwBrowserHAB;  INACTIVE
-//CWBrowserAK cwBrowserAK;    INACTIVE
-//CWBrowserSA cwBrowserSA;    INACTIVE
-CWBrowserWW180 cwBrowserWW180;
-CWBrowserWW360 cwBrowserWW360;
-CWDataBrowser cwDataBrowser;
 dods.dap.DConnect dConnect;
 dods.dap.DFloat64 dFloat64;
 dods.dap.DInt16 dInt16;
@@ -853,45 +842,31 @@ public static void main(String args[]) throws Throwable {
 dods.dap.DSequence dseq;
 DoubleArray doublea;
 EDDTableFromAllDatasets etfad;
-EmaAttribute ea;
-EmaClass ec;
-EmaColor ecolor;
 File2 f2;
 FileNameUtility fnu;
 FileVisitorDNLS fvdnls;
 FileVisitorSubdir fvsd;
 FilledMarkerRenderer fmr;
 FloatArray floata;
-GDateTime gdt;
 GenerateThreddsXml gtx;
 GraphDataLayer gdl;
 Grid grid;
-GridDataSet gds;
-GridDataSetAnomaly gdsa;
-GridDataSetOpendap gdso;
-GridDataSetThredds gdst;
-GridScreen gs;
 GSHHS gshhs;
 HashDigest hd;
 Image2 i2;
 IntArray inta;
 JSONObject jo;
 org.json.JSONTokener jt;
 LongArray la;
-MapScreen mapScreen;
 Math2 m2;
 Matlab matlab;     
 MustBe mb;
 NcHelper ncHelper;
 NetCheck netCheck;
-OneOf oneOf;
 OpendapHelper opendapHelper;
 PAOne paOne;
-ParseJSON parseJSON;
 PauseTest pt;
 PlainAxis2 sgtpa2;
-PointScreen ps;
-PointVectorScreen pvs;
 PrimitiveArray primitiveArray; 
 RegexFilenameFilter rff;
 ResourceBundle2 rb2;
@@ -902,7 +877,6 @@ public static void main(String args[]) throws Throwable {
 SgtGraph sgtGraph; 
 SgtMap sgtMap;     
 SgtUtil sgtUtil;
-Shared shared;
 gov.noaa.pfel.coastwatch.sgt.PathCartesianRenderer sgtptcr;
 gov.noaa.pmel.sgt.AnnotationCartesianRenderer sgtacr;
 gov.noaa.pmel.sgt.AxisTransform sgtat;
@@ -965,15 +939,13 @@ public static void main(String args[]) throws Throwable {
 Tally tally;
 Test test;
 Touch touch;
-TrajectoryScreen trajs;
 UByteArray uba;
 UIntArray uia;
 ULongArray ula;
 Units2 u2;
 UShortArray usa;
 gov.noaa.pmel.sgt.VectorCartesianRenderer vcr;
 VectorPointsRenderer vpr;
-VectorScreen vs;
 WatchDirectory wdir;
 XML xml;
 
@@ -1042,7 +1014,6 @@ public static void main(String args[]) throws Throwable {
 Erddap erddap;       
 ErddapRedirect erddapRedirect;       
 FindDuplicateTime findDuplicateTime;
-FishBase fb;
 GenerateDatasetsXml gdx;
 GridDataAccessor gda;
 GridDataAllAccessor gdaacc;
@@ -1057,7 +1028,6 @@ public static void main(String args[]) throws Throwable {
 OutputStreamFromHttpResponseViaAwsS3 osfhrvas;
 OutputStreamViaAwsS3 osvas;
 PersistentTable pert;
-Projects2 proj2;
 RunLoadDatasets rld;
 
 Subscriptions sub;

WEB-INF/classes/gov/noaa/pfel/coastwatch/TimePeriods.properties

@@ -0,0 +1,96 @@
+# TimePeriods.properties contains information needed for FileNameUtility.
+# Properties files have info in the form: name = value
+# Spaces before and after the "=" are ignored.
+# Backslash, \, can be used at the end of a line in a .properties
+#   file to indicate that the logical line is continued on the next physical line.
+# The formfeed character, \f, is also often used as a separator.
+# Sometimes the left single quote character, `, is used as a lower-level separator 
+#   when a tab might be used in a Unix document. But tabs are hard to work with
+#   since you can't actually see them.
+# Lines starting with "#" are comments.
+
+# *****************************************************************************
+
+### The rest of the file contains keys/values for EmaAttributes
+### which are members of this class and related information used by CWBrowser.
+
+# Info for the 'Region' option on the Map screen
+# Regions must be from large -> small areas so image map can detect small areas
+#   which are within larger areas.
+region.options = \
+    US+Mexico\f\
+    West US\f\
+    N\f\
+    N1\f\
+    N2\f\
+    N3\f\
+    C\f\
+    C1\f\
+    C2\f\
+    C3\f\
+    AN\f\
+    BB\f\
+    GG\f\
+    MB\f\
+    SF\f\
+    S\f\
+    S1\f\
+    S2\f\
+    M\f\
+    M1\f\
+    M2
+
+# For each region above, define a title for its radio button.
+region.title = You can describe a custom region at left, or select one of the pre-defined regions here or by clicking on the map at right.\f\
+    US+Mexico: Click here to view a map of the entire U.S. west coast and most of the west coast of Mexico.\f\
+    West US: Click here to view a map of the entire U.S. west coast.\f\
+    N: Click here to view a map of the Washington and Oregon coast; NANOOS (Northwest Association of Networked Ocean Observing Systems).\f\
+    N1: Click here to view a map of the Washington coast (northern part of NANOOS: NW01).\f\
+    N2: Click here to view a map of the Washington and Oregon coast (central part of NANOOS: NW02).\f\
+    N3: Click here to view a map of the Oregon coast (southern part of NANOOS: NW03).\f\
+    C: Click here to view a map of the Northern and central California coast; CeNCOOS (Central California Ocean Observing System).\f\
+    C1: Click here to view a map of the Northern California coast (northern part of CenCOOS: CW01).\f\
+    C2: Click here to view a map of the California coast north of S.F. (central part of CenCOOS:CW02).\f\
+    C3: Click here to view a map of the California coast just south of S.F. (southern part of CenCOOS: CW03).\f\
+    AN: Click here to view a map of Ano Nuevo.\f\
+    BB: Click here to view a map of Bodega Bay.\f\
+    GG: Click here to view a map of west of the Golden Gate Bridge.\f\
+    MB: Click here to view a map of the Monterey Bay.\f\
+    SF: Click here to view a map of the San Francisco Bay.\f\
+    S: Click here to view a map of the Southern California coast; SCCOOS (Southern California Coastal Ocean Observing System).\f\
+    S1: Click here to view a map of the California coast just north of L.A. (northern part of SCCOOS: SW01).\f\
+    S2: Click here to view a map of the California coast south of L.A. (southern part of SCCOOS: SW02).\f\
+    M: Click here to view a map of the Northern and central west coast of Mexico.\f\
+    M1: Click here to view a map of the Northern west coast of Mexico.\f\
+    M2: Click here to view a map of the Central west coast of Mexico.
+
+# For each region above, define the range 
+# Items: rectangleARGBColor, minX, maxX, minY, maxY, LabelLeftX, LabelBottomY, labelText 
+# x,y are lower left lon and lat in decimal degrees.  
+# Another good color is orange: ffcc00
+# The first range determines the range of the coastline boundaries which are 
+#    pre-loaded; so the first range must encompass all other ranges.
+regionInfo = \
+    0x00FFFFFF, -135,   -105,   22,   51,   -131,    25,    US+Mexico\f\
+    0x306666CC, -135,   -113,   29,   51,   -133,    31,    West US\f\
+    0x30ff00ff, -131,   -122,   41,   51,   -129.75, 46,    N\f\
+    0x30ff00ff, -126,   -122.5, 46.5, 51,   -128,    47.75, N1\f\
+    0x30ff00ff, -126.5, -123.5, 44.5, 47.5, -126.5,  45.75, N2\f\
+    0x30ff00ff, -127.5, -124,   42,   45.5, -127,    43,    N3\f\
+    0x300000ff, -129.5, -120.5, 33.5, 42.5, -128,    37,    C\f\
+    0x300000ff, -126.5, -123,   38.5, 42,   -126.5,  39.5,  C1\f\
+    0x300000ff, -125,   -121,   35,   39,   -125.25, 36.5,  C2\f\
+    0x300000ff, -123,   -120,   34,   37,   -124,    35,    C3\f\
+    0x300000ff, -123.3, -122,   36.5, 37.8, -122.5,  36.75, AN\f\
+    0x300000ff, -123.85,-122.8, 37.75,38.5, -122.75, 38,    BB\f\
+    0x300000ff, -123.5, -122.15,36.95,38.05,-124,    37.5,  GG\f\
+    0x300000ff, -123.1, -121.6, 35.9, 37.4, -121.5,  36.25, MB\f\
+    0x300000ff, -122.47,-122.3, 37.8, 37.95,-121.75, 37.25, SF\f\
+    0x3000ffff, -125,   -116.5, 29,   35,   -123.25, 30,    S\f\
+    0x3000ffff, -122,   -119,   32,   35,   -122,    32.5,  S1\f\
+    0x3000ffff, -119.5, -116.5, 31.5, 34.5, -119.75, 31.5,  S2\f\
+    0x3000ff00, -120,   -105,   22,   33,   -118.75, 23,    M\f\
+    0x3000ff00, -118,   -110,   27,   32,   -117.75, 28,    M1\f\
+    0x3000ff00, -116,   -105,   22,   28,   -115,    23,    M2
+
+### end of file

WEB-INF/classes/gov/noaa/pfel/coastwatch/griddata/Grid.java

@@ -57,12 +57,6 @@
  */
 public class Grid  {
 
-    //ensure org.jdom.Content is compiled -- 
-    //GeotiffWriter needs it, but it isn't called directly so
-    //it isn't automatically compiled.
-    private org.jdom.Content content;
-
-
     /** A 1D array, column by column, from the lower left (the way SGT wants it). 
         Missing values are stored as NaN's.        
         Note that this could/should be a PrimitiveArray (to conserve memory), but

WEB-INF/classes/gov/noaa/pfel/coastwatch/griddata/GridSaveAs.java

@@ -29,7 +29,7 @@ public class GridSaveAs {
      */
     public static void main(String args[]) throws Exception {
         //Grid.verbose = true;
-        Grid.davesSaveAs(args, new FileNameUtility("gov.noaa.pfel.coastwatch.CWBrowser"));
+        Grid.davesSaveAs(args, new FileNameUtility("gov.noaa.pfel.coastwatch.TimePeriods"));
 
     }
 

WEB-INF/classes/gov/noaa/pfel/coastwatch/util/FileVisitorDNLS.java

@@ -7,6 +7,7 @@
 import com.cohort.array.Attributes;
 import com.cohort.array.DoubleArray;
 import com.cohort.array.LongArray;
+import com.cohort.array.PrimitiveArray;
 import com.cohort.array.StringArray;
 import com.cohort.util.Calendar2;
 import com.cohort.util.File2;
@@ -2396,12 +2397,16 @@ public static void reduceDnlsTableToOneDir(Table dnlsTable, String oneDir, HashS
         if (nRows == 0)
             return;
         char separator = oneDir.indexOf('\\') >= 0? '\\' : '/';
+        char unusedSeparator = oneDir.indexOf('\\') >= 0? '/' : '\\';
+        oneDir = oneDir.replace(unusedSeparator, separator);
         StringArray dirSA  = (StringArray)dnlsTable.getColumn(0);
         StringArray nameSA = (StringArray)dnlsTable.getColumn(1);
         int oneDirLength = oneDir.length();
         BitSet keep = new BitSet(nRows);  //all false
         for (int row = 0; row < nRows; row++) {
             String tDir = dirSA.get(row);
+            // Make sure the separator in tDir matches the separator used in oneDir.
+            tDir = tDir.replace(unusedSeparator, separator);
             if (tDir.startsWith(oneDir)) {
                 if (tDir.length() == oneDirLength) {
                     if (nameSA.get(row).length() > 0) 
@@ -2417,6 +2422,25 @@ public static void reduceDnlsTableToOneDir(Table dnlsTable, String oneDir, HashS
         //String2.log(">> reduceDnlsTabletoOneDir nRows=" + dnlsTable.nRows() + " nSubdir=" + subdirHash.size());
     }
 
+    public static int indexOfDirectory(PrimitiveArray directories, String toMatch) {
+        int dirIndex = directories.indexOf(toMatch);
+        if (dirIndex >= 0) {
+            return dirIndex;
+        }
+        char separator = toMatch.indexOf('\\') >= 0? '\\' : '/';
+        char unusedSeparator = toMatch.indexOf('\\') >= 0? '/' : '\\';
+        toMatch = toMatch.replace(unusedSeparator, separator);
+
+        for (int row = 0; row < directories.size(); row++) {
+            String dir = directories.getString(row);
+            dir = dir.replace(unusedSeparator, separator);
+            if (toMatch.equals(dir)) {
+                dirIndex = row;
+            }
+        }
+        return dirIndex;
+    }
+
    /**
      * This is used for testing this class.
      * This is used when called from the command line.

WEB-INF/classes/gov/noaa/pfel/coastwatch/util/SSR.java

@@ -254,7 +254,7 @@ public static String getFirstLineMatching(String fileName, String charset, Strin
      */
     public static ArrayList dosOrCShell(String commandLine, int timeOutSeconds) throws Exception {
         if (String2.OSIsWindows) {
-            commandLine = String2.replaceAll(commandLine, "/", "\\");
+            // commandLine = String2.replaceAll(commandLine, "/", "\\");
             return dosShell(commandLine, timeOutSeconds);  
         } else {
             return cShell(  commandLine, timeOutSeconds);  

WEB-INF/classes/gov/noaa/pfel/erddap/dataset/EDDGrid.java

@@ -323,11 +323,6 @@ public abstract class EDDGrid extends EDD {
 
     }
 
-    //ensure org.jdom.Content is compiled -- 
-    //GeotiffWriter needs it, but it isn't called directly so
-    //it isn't automatically compiled.
-    private static org.jdom.Content orgJdomContent;
-
     //the diagnostic tests change this just for testing
     static int tableWriterNBufferRows = 100000;
 

WEB-INF/classes/gov/noaa/pfel/erddap/dataset/EDDGridFromFiles.java

@@ -1983,7 +1983,7 @@ public String accessibleViaFilesGetLocal(int language, String relativeFileName)
             Table dirTable  = getDirTable();  //no need to get copy since not changing it
             Table fileTable = getFileTable(); //no need to get copy since not changing it 
             PrimitiveArray dirNames = dirTable.getColumn(0); //the only column
-            int dirIndex = dirNames.indexOf(localDir);
+            int dirIndex = FileVisitorDNLS.indexOfDirectory(dirNames, localDir);
             if (dirIndex < 0) {
                 String2.log(msg + "localDir=" + localDir + " not in dirTable.");
                 return null;

WEB-INF/classes/gov/noaa/pfel/erddap/dataset/EDDTableFromFileNames.java

@@ -5,7 +5,6 @@
 package gov.noaa.pfel.erddap.dataset;
 
 import com.cohort.array.Attributes;
-import com.cohort.array.ByteArray;
 import com.cohort.array.DoubleArray;
 import com.cohort.array.LongArray;
 import com.cohort.array.PAOne;
@@ -28,27 +27,17 @@
 import gov.noaa.pfel.coastwatch.util.SSR;
 
 import gov.noaa.pfel.erddap.Erddap;
-import gov.noaa.pfel.erddap.GenerateDatasetsXml;
 import gov.noaa.pfel.erddap.util.EDStatic;
 import gov.noaa.pfel.erddap.variable.*;
 
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.FileWriter;
-import java.io.StringWriter;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.BitSet;
-import java.util.Enumeration;
-import java.util.GregorianCalendar;
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.regex.Matcher;
 import java.util.regex.Pattern;
 
-//import org.apache.commons.jexl3.introspection.JexlSandbox;
-import org.apache.commons.jexl3.JexlScript;
-import org.apache.commons.jexl3.MapContext;
 
 /** 
  * This class represents a table of file names.