HADOOP-19762: Move hadoop-gcp from hadoop-tools to hadoop-cloud-storage-project

[cnauroth]

Description of PR

When we introduced hadoop-gcp, we discussed whether this belongs under hadoop-cloud-storage-project or hadoop-tools in a comment chain starting here:

https://issues.apache.org/jira/browse/HADOOP-19343?focusedCommentId=17941734&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17941734

We chose hadoop-tools, but in retrospect, hadoop-cloud-storage-project is really the right choice. hadoop-aws and hadoop-azure remain under hadoop-tools mostly for legacy reasons. Moving them under hadoop-cloud-storage-project would cause a lot of maintenance burden while backporting patches to prior versions. For a new, unreleased cloud FileSystem, we have the opportunity to ship it in hadoop-cloud-storage-project from the start.

1. git mv hadoop-tools/hadoop-gcp hadoop-cloud-storage-project
2. Remove hadoop-gcp module from hadoop-tools/pom.xml and add to hadoop-cloud-storage-project/pom.xml.
3. Small modification in hadoop-gcp/pom.xml to the way maven-dependency-plugin:list is called for shell profile generation.
4. Reorder the module list in the top-level pom.xml so that hadoop-dist is always last. Otherwise, hadoop-dist runs the dist-layout-stitching script before hadoop-gcp has run its shading in the package phase, so it accidentally picks up the unshaded jar.

How was this patch tested?

1. Create a distro including hadoop-gcp:

mvn clean package -Pdist -Dtar -Dhadoop-gcp-package -DskipTests

2. Extract:

tar xf ~/hadoop/hadoop-dist/target/hadoop-3.5.0-SNAPSHOT.tar.gz

3. Confirm inclusion of hadoop-gcp.jar:

ls -lrth hadoop-3.5.0-SNAPSHOT/share/hadoop/common/lib/hadoop-gcp-3.5.0-SNAPSHOT.jar 
-rw-r--r-- 1 cnauroth cnauroth 34M Dec 26 23:09 hadoop-3.5.0-SNAPSHOT/share/hadoop/common/lib/hadoop-gcp-3.5.0-SNAPSHOT.jar

4. Test a command accessing GCS:

bin/hadoop fs \
    -D fs.gs.auth.type=SERVICE_ACCOUNT_JSON_KEYFILE \
    -D fs.gs.auth.service.account.json.keyfile=/usr/local/google/home/cnauroth/cnauroth-test.json \
    -ls gs://cnauroth-hive-metastore-proxy-dist

5. Run hadoop-gcp integration tests:

mvn verify

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[cnauroth]

Hello @steveloughran and @mukund-thakur . Can you please review?

@arunkumarchacko FYI.