Release 2022-09-14. Bandwidth enhancements.

The bandwidth command now
* can save details to a CSV file
* adds options to set: read policy, block size, whether to flush/hflush after each write
* reports whether progress callbacks were made during (possibly slow) close() operations.

Read the bandwidth documentation for details and example analysis of CSV files using
different S3A tuning parameters (including prefetching).

Author: steveloughran

BUILDING.md

@@ -19,10 +19,12 @@ With maven, with profiles for many different hadoop versions.
 To build a production release
 1. Use java8
 2. And compile against a shipping hadoop version, with `-Pextra` for the extra stuff
+3. Do not attempt to build the AWS v1 SDK components `-Pextra` against a version of Hadoop
+   built with the AWS V2 SDK. It will fail
 
 
 ```bash
-mvn clean install -Phadoop-3.3.2 -Pextra
+mvn clean install -Pextra
 ```
 
 ## Releasing

README.md

@@ -25,17 +25,17 @@ kept out right now for various reasons
 Why? 
 
 1. Sometimes things fail, and the first problem is classpath;
-1. The second, invariably some client-side config. 
-1. Then there's networking and permissions...
-1. The Hadoop FS connectors all assume a well configured system, and don't
+2. The second, invariably some client-side config. 
+3. Then there's networking and permissions...
+4. The Hadoop FS connectors all assume a well configured system, and don't
 do much in terms of meaningful diagnostics.
-1. This is compounded by the fact that we dare not log secret credentials.
-1. And in support calls, it's all to easy to get those secrets, even
+5. This is compounded by the fact that we dare not log secret credentials.
+6. And in support calls, it's all to easy to get those secrets, even
 though its a major security breach to get them.
 
 ### Secondary: higher performance cloud IO
 
-The main hadoop `hadoop fs` commands are written assuming a filesystem, where
+The main hadoop `hadoop fs` commands are written assuming a filesystem, where:
 
 * Recursive treewalks are the way to traverse the store.
 * The code was written for Hadoop 1.0 and uses the filesystem APIs of that era.
@@ -133,16 +133,7 @@ The comments are printed too! This means you can use them in the reports.
 
 ## Command `bandwidth`
 
-Measure upload/download bandwidth.
-
-```bash
- bin/hadoop jar cloudstore-1.0.jar bandwidth
-Usage: bandwidth [options] size <path>
-        -D <key=value>  Define a property
-        -tokenfile <file>       Hadoop token file to load
-        -verbose        print verbose output
-        -xmlfile <file> XML config file to load
-```
+Measure upload/download bandwidth, optionally saving data to a CSV file.
 
 See [bandwidth](src/main/site/bandwidth.md) for details.
 

src/main/java/org/apache/hadoop/fs/store/CommonParameters.java

@@ -29,9 +29,6 @@ public final class CommonParameters {
   /** {@value}. */
   public static final String XMLFILE = "xmlfile";
 
-  /** {@value}. */
-  public static final String CSVFILE = "csv";
-
   /** file for system properties {@value}. */
   public static final String SYSPROPFILE = "syspropfile";
 

src/main/java/org/apache/hadoop/fs/store/StoreDurationInfo.java

@@ -38,6 +38,8 @@ public class StoreDurationInfo
 
   private long finished;
 
+  private boolean isFinished;
+
   private final String text;
 
   private final Logger log;
@@ -78,7 +80,7 @@ public StoreDurationInfo(PrintStream out, String format, Object... args) {
     }
   }
 
- /**
+  /**
    * Create the duration with no output printed.
    */
   public StoreDurationInfo() {
@@ -93,8 +95,14 @@ private long time() {
     return System.currentTimeMillis();
   }
 
-  public void finished() {
-    finished = time();
+  /**
+   * Finish the operation; only valid once.
+   */
+  public synchronized void finished() {
+    if (!isFinished) {
+      finished = time();
+      isFinished = true;
+    }
   }
 
   public String getDurationString() {
@@ -104,7 +112,7 @@ public String getDurationString() {
   public static String humanTime(long time) {
     long seconds = (time / 1000);
     long minutes = (seconds / 60);
-    return String.format("%d:%02d:%03d", minutes, seconds % 60, time % 1000);
+    return String.format("%d:%02d.%03d", minutes, seconds % 60, time % 1000);
   }
 
   /**
@@ -122,11 +130,17 @@ public long value() {
   public Duration asDuration() {
     return Duration.ofMillis(value());
   }
+
   @Override
   public String toString() {
     return getDurationString();
   }
 
+
+  /**
+   * Close the duration, invoke {@link #finished()}.
+   * Log the final state if a log or output stream was provided.
+   */
   @Override
   public void close() {
     finished();

src/main/java/org/apache/hadoop/fs/store/StoreEntryPoint.java

@@ -155,7 +155,6 @@ public final void println() {
   /**
    * Print a formatted string followed by a newline to the output stream.
    * @param format format string
-   * @param args optional arguments
    */
   @Override
   public void println(String format) {
@@ -297,6 +296,16 @@ protected final String getOption(String opt) {
     return getCommandFormat().getOptValue(opt);
   }
 
+  /**
+   * Get the value of a key-val option.
+   * @param opt option.
+   * @param defval default value
+   * @return the value or null
+   */
+  protected final String getOption(String opt, String defval) {
+    return hasOption(opt) ? getCommandFormat().getOptValue(opt) : defval;
+  }
+
   /**
    * Did the command line have a specific option.
    * @param opt option.