2022-06-16 awssdk_config_override.json

s3a storediag adds two optional resources and prints their locations, if found

com/amazonaws/internal/config/awssdk_config_default.json
awssdk_config_override.json

The awssdk_config_default.json file is probably required, we are just being
cautious.

2022-06-15 release: public-suffix-list awareness

This release is related to HADOOP-18159 Certificate doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]

for s3a and gs, it will now print which location the shaded/unshaded httpclient is picking up its list of toplevel domains from.

This is determined by loading the resource mozilla/public-suffix-list.txt on the classpath.

If an out of date version of this file is found first on the classpath, the clients may fail to talk to the stores.

note: the location of the resource can also be found by

hadoop org.apache.hadoop.util.FindClass locate mozilla/public-suffix-list.txt

2022-06-09-release: listobjects

listobject tuning

• limit command to fail faster
• only call getFileStatus if -verbose is true.

makes things faster when working with large directory trees

2022-06-06 release

ongoing maintenance work

2022-04-20 release: class diagnostics improved

storediag prints resource of each class before loading it

this identifies the location of a class even if class load fails.

2022-04-20 release: performance hints

Adding some analysis of options and performance hints, initially just for the s3a connector

example

Performance Hints
=================

Option fs.s3a.threads.max is unset. Recommend a value of at least 512
Option fs.s3a.connection.maximum is unset. Recommend a value of at least 1024
Option fs.s3a.committer.threads is unset. Recommend a value of at least 256

2022-04-12 maintenance

maintenance release; nothing of importance

2022-03-30 more options and env vars printed

a bit more printing of things, primarily to debug TLS/openssl issues

• prints fs.creation.parallel.count
• "LD_LIBRARY_PATH" env var
• java version sysprop

2022-03-22 TLS diagnostics edition

storediag to print TLS info

• all jdk sysprops on the topic in the docs
• enumerates available encryption protocols
  (doesn't go into the details of algorithms tho')

Also, add a counter to all enumerated values (configs, properties, env vars)...

2022-03-manifest-committer

MAPREDUCE-7341. gcs and abfs committer options printed.

• s3a diag looks more at endpoints and regions
• println does %n to newline mapping everywhere