ODP-6516 : Add util for postinstall 3.3.6.3-101 ambari_java home related fixes

util-3.3.6.3-101-ambari_java_home/.gitignore

@@ -0,0 +1 @@
+/backups/

util-3.3.6.3-101-ambari_java_home/README.md

@@ -0,0 +1,42 @@
+# util-3.3.6.3-101-ambari_java_home
+
+Ambari Server utility for **ODP 3.3** (e.g. **3.3.6.3-101**): applies **[ODP-6189](https://github.com/acceldata-io/odp-ambari/pull/484)** so **CredentialUtil** uses **`ambari_java_home`** / **`$AMBARI_JAVA_HOME/bin/java`** instead of **`java`** on `PATH` (avoids failures when the stack uses Java 11).
+
+**Does:** (1) copies three vendored **`files/stacks/ODP/3.3/.../*.py`** into **`/var/lib/ambari-server/resources/...`** with backups under **`$BACKUP_ROOT/<timestamp>/`**; (2) **`configs.py` get → sed/awk on `kafka-env` / `cruise-control-env` `content` → set** (skips set if nothing changed). Helpers under **`lib/`**: **`ambari_cluster_name.py`** (autodiscover `CLUSTER` via REST), **`json_content_roundtrip.py`**, **`file_sha256.py`**. Stack definition **XML** is not applied here; see **`temp.md`** if you need that diff manually.
+
+## Run
+
+**Host:** must be the **Ambari Server** node (script checks for **`/var/lib/ambari-server`**). **`AMBARI_PORT`** defaults to **8080**; must be **1–65535** if set. **`AMBARI_PROTOCOL`** is **`http`** or **`https`** (case-insensitive).
+
+```bash
+export AMBARI_USER=admin AMBARI_PASSWORD='***'
+export CLUSTER=mycluster   # optional if API infers one cluster
+# export AMBARI_HOST=$(hostname -f) AMBARI_PORT=8443 AMBARI_PROTOCOL=https
+sudo -E ./patch_ambari_java_home.sh
+```
+
+| Flag | |
+|------|--|
+| `--no-stack-python` | Skip `files/` → resources copy |
+| `--no-cluster-config` | Skip `configs.py` |
+| `--dry-run` | Stack: compare only. Cluster: no `configs.py set` |
+
+**Needs:** root for stack copies; **`configs.py`** at **`$AMBARI_RESOURCES/scripts/configs.py`**. **`PYTHON_BIN`** defaults to **`python3.11`**. Override **`CONFIGS_PYTHON_BIN`** only if **`configs.py`** must use another interpreter. With **`AMBARI_PROTOCOL=https`**, **`configs.py`** is called with **`--unsafe`** (skip TLS cert verify) unless you set **`AMBARI_SSL_VERIFY_STRICT=1`**.
+
+### After this util
+
+1. **Cluster configs** (`kafka-env`, `cruise-control-env`): Ambari stores the new version; agents use it the next time those components run **install / configure / start / restart**. Do a **rolling restart** (or restart) of **Kafka**, **Cruise Control**, and **Druid** (and any other consumer of those configs).
+
+2. **Stack `*.py`** on the server (`/var/lib/ambari-server/resources/stacks/...`): agents often keep **cached** copies under **`/var/lib/ambari-agent/`** (exact layout varies by Ambari/ODP). Until the cache is refreshed, a host might still run **old** `params.py` / `kafka.py` for stack-driven actions.
+
+   **Recommended:** on **each host** that runs Kafka, Cruise Control, Druid (or simply **all** cluster nodes), as root:
+
+   ```bash
+   sudo ambari-agent restart
+   ```
+
+   That restarts the agent and typically rebuilds cache from the server on the next command.
+
+3. **If something still looks stale** (rare): with the agent **stopped**, only after checking your Ambari/ODP version docs, some sites clear **`/var/lib/ambari-agent/cache`** (or the version-specific cache tree) then start the agent again — **do not** delete arbitrary paths without confirming with your runbook.
+
+4. Optionally **restart Ambari Server** if your operations guide requires it after editing `resources/` (often **not** strictly required for stack file edits alone).

util-3.3.6.3-101-ambari_java_home/files/stacks/ODP/3.3/services/DRUID/package/scripts/params.py

@@ -0,0 +1,225 @@
+#!/usr/bin/env ambari-python-wrap
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+from ambari_commons import OSCheck
+from resource_management.libraries.functions import conf_select
+from resource_management.libraries.functions import stack_select
+from resource_management.libraries.functions.get_bare_principal import get_bare_principal
+from resource_management.libraries.resources.hdfs_resource import HdfsResource
+from resource_management.libraries.functions import get_kinit_path
+from resource_management.libraries.script.script import Script
+from resource_management.libraries.functions.expect import expect
+from resource_management.libraries.functions import format
+from resource_management.libraries.functions.get_not_managed_resources import get_not_managed_resources
+from resource_management.libraries.functions.default import default
+from resource_management.libraries.functions.lzo_utils import should_install_lzo
+from ambari_commons.constants import AMBARI_SUDO_BINARY
+import status_params
+
+# a map of the Ambari role to the component name
+# for use with <stack-root>/current/<component>
+SERVER_ROLE_DIRECTORY_MAP = {
+  'DRUID_BROKER': 'druid-broker',
+  'DRUID_COORDINATOR': 'druid-coordinator',
+  'DRUID_HISTORICAL': 'druid-historical',
+  'DRUID_MIDDLEMANAGER': 'druid-middlemanager',
+  'DRUID_OVERLORD': 'druid-overlord',
+  'DRUID_ROUTER': 'druid-router'
+}
+
+# server configurations
+config = Script.get_config()
+stack_root = Script.get_stack_root()
+tmp_dir = Script.get_tmp_dir()
+
+stack_name = default("/clusterLevelParams/stack_name", None)
+
+# stack version
+stack_version = default("/commandParams/version", None)
+
+# un-formatted stack version
+stack_version_unformatted = str(config['clusterLevelParams']['stack_version'])
+
+# default role to coordinator needed for service checks
+component_directory = Script.get_component_from_role(SERVER_ROLE_DIRECTORY_MAP, "DRUID_COORDINATOR")
+
+hostname = config['agentLevelParams']['hostname']
+sudo = AMBARI_SUDO_BINARY
+
+# default druid parameters
+druid_home = format("{stack_root}/current/{component_directory}")
+druid_conf_dir = format("{stack_root}/current/{component_directory}/conf")
+
+druid_common_conf_dir = druid_conf_dir + "/_common"
+druid_coordinator_conf_dir = druid_conf_dir + "/coordinator"
+druid_overlord_conf_dir = druid_conf_dir + "/overlord"
+druid_broker_conf_dir = druid_conf_dir + "/broker"
+druid_historical_conf_dir = druid_conf_dir + "/historical"
+druid_middlemanager_conf_dir = druid_conf_dir + "/middleManager"
+druid_router_conf_dir = druid_conf_dir + "/router"
+druid_extensions_dir = druid_home + "/extensions"
+druid_hadoop_dependencies_dir = druid_home + "/hadoop-dependencies"
+druid_segment_infoDir = config['configurations']['druid-historical']['druid.segmentCache.infoDir']
+druid_segment_cache_locations = config['configurations']['druid-historical']['druid.segmentCache.locations']
+druid_tasks_dir = config['configurations']['druid-middlemanager']['druid.indexer.task.baseTaskDir']
+druid_user = config['configurations']['druid-env']['druid_user']
+druid_log_dir = config['configurations']['druid-env']['druid_log_dir']
+druid_classpath = config['configurations']['druid-env']['druid_classpath']
+druid_extensions = config['configurations']['druid-common']['druid.extensions.pullList']
+druid_repo_list = config['configurations']['druid-common']['druid.extensions.repositoryList']
+druid_extensions_load_list = config['configurations']['druid-common']['druid.extensions.loadList']
+druid_security_extensions_load_list = config['configurations']['druid-common']['druid.security.extensions.loadList']
+
+
+# status params
+druid_pid_dir = status_params.druid_pid_dir
+user_group = config['configurations']['cluster-env']['user_group']
+java8_home = config['ambariLevelParams']['java_home']
+java_version = expect("/ambariLevelParams/java_version", int)
+# Use Ambari's Java home for DBConnectionVerification.jar (compiled with JDK17+)
+ambari_java_home = default("/ambariLevelParams/ambari_java_home", java8_home)
+druid_env_sh_template = config['configurations']['druid-env']['content']
+
+# jceks params
+jceks_path = "jceks://file/etc/security/credential/druid.jceks"
+# Use Ambari's Java home for CredentialUtil (compiled with JDK17+)
+password_command = "{0}/bin/java -cp '/var/lib/ambari-agent/cred/lib/*' org.apache.ambari.server.credentialapi.CredentialUtil -provider {1} get ".format(ambari_java_home, jceks_path)
+
+# log4j params
+log4j_props = config['configurations']['druid-log4j']['content']
+druid_log_level = config['configurations']['druid-log4j']['druid_log_level']
+metamx_log_level = config['configurations']['druid-log4j']['metamx_log_level']
+root_log_level = config['configurations']['druid-log4j']['root_log_level']
+
+druid_log_maxbackupindex = default('/configurations/druid-logrotate/druid_log_maxbackupindex', 7)
+druid_log_maxfilesize = default('/configurations/druid-logrotate/druid_log_maxfilesize', 256)
+logrotate_props = config['configurations']['druid-logrotate']['content']
+
+# Metadata storage
+metadata_storage_user = config['configurations']['druid-common']['druid.metadata.storage.connector.user']
+metadata_storage_password = config['configurations']['druid-common']['druid.metadata.storage.connector.password']
+metadata_storage_db_name = config['configurations']['druid-common']['database_name']
+metadata_storage_db_name = config['configurations']['druid-common']['database_name']
+metadata_storage_type = config['configurations']['druid-common']['druid.metadata.storage.type']
+metadata_storage_url = config['configurations']['druid-common']['druid.metadata.storage.connector.connectURI']
+jdk_location = config['ambariLevelParams']['jdk_location']
+if 'mysql' == metadata_storage_type:
+  jdbc_driver_jar = default("/ambariLevelParams/custom_mysql_jdbc_name", None)
+  connector_curl_source = format("{jdk_location}/{jdbc_driver_jar}")
+  connector_download_dir=format("{druid_extensions_dir}/mysql-metadata-storage")
+  downloaded_custom_connector = format("{tmp_dir}/{jdbc_driver_jar}")
+
+check_db_connection_jar_name = "DBConnectionVerification.jar"
+check_db_connection_jar = format("/usr/lib/ambari-agent/{check_db_connection_jar_name}")
+
+# HDFS
+security_enabled = config['configurations']['cluster-env']['security_enabled']
+hdfs_user = config['configurations']['hadoop-env']['hdfs_user']
+kinit_path_local = get_kinit_path(default('/configurations/kerberos-env/executable_search_paths', None))
+hdfs_user_keytab = config['configurations']['hadoop-env']['hdfs_user_keytab']
+hadoop_bin_dir = stack_select.get_hadoop_dir("bin")
+hadoop_conf_dir = conf_select.get_hadoop_conf_dir()
+hdfs_principal_name = default('/configurations/hadoop-env/hdfs_principal_name', 'missing_principal').replace("_HOST",
+                                                                                                             hostname)
+hdfs_site = config['configurations']['hdfs-site']
+default_fs = config['configurations']['core-site']['fs.defaultFS']
+dfs_type = default("/clusterLevelParams/dfs_type", "")
+hdfs_tmp_dir = config['configurations']['hadoop-env']['hdfs_tmp_dir']
+
+# Kerberos
+druid_principal_name = default('/configurations/druid-common/druid.hadoop.security.kerberos.principal',
+                               'missing_principal')
+druid_user_keytab = default('/configurations/druid-common/druid.hadoop.security.kerberos.keytab', 'missing_keytab')
+
+import functools
+
+# create partial functions with common arguments for every HdfsResource call
+# to create hdfs directory we need to call params.HdfsResource in code
+HdfsResource = functools.partial(
+  HdfsResource,
+  user=hdfs_user,
+  hdfs_resource_ignore_file="/var/lib/ambari-agent/data/.hdfs_resource_ignore",
+  security_enabled=security_enabled,
+  keytab=hdfs_user_keytab,
+  kinit_path_local=kinit_path_local,
+  hadoop_bin_dir=hadoop_bin_dir,
+  hadoop_conf_dir=hadoop_conf_dir,
+  principal_name=hdfs_principal_name,
+  hdfs_site=hdfs_site,
+  default_fs=default_fs,
+  immutable_paths=get_not_managed_resources(),
+  dfs_type=dfs_type
+)
+
+# Ambari Metrics
+metric_emitter_type = "noop"
+metric_collector_host = ""
+metric_collector_port = ""
+metric_collector_protocol = ""
+metric_truststore_path = default("/configurations/ams-ssl-client/ssl.client.truststore.location", "")
+metric_truststore_type = default("/configurations/ams-ssl-client/ssl.client.truststore.type", "")
+metric_truststore_password = default("/configurations/ams-ssl-client/ssl.client.truststore.password", "")
+
+ams_collector_hosts = default("/clusterHostInfo/metrics_collector_hosts", [])
+if 'cluster-env' in config['configurations'] and \
+    'metrics_collector_external_hosts' in config['configurations']['cluster-env']:
+  ams_collector_hosts = config['configurations']['cluster-env']['metrics_collector_external_hosts']
+  set_instanceId = "true"
+else:
+  ams_collector_hosts = ",".join(default("/clusterHostInfo/metrics_collector_hosts", []))
+has_metric_collector = not len(ams_collector_hosts) == 0
+
+if has_metric_collector:
+  metric_emitter_type = "ambari-metrics"
+  metric_collector_host = ams_collector_hosts[0]
+  if 'cluster-env' in config['configurations'] and \
+      'metrics_collector_external_port' in config['configurations']['cluster-env']:
+    metric_collector_port = config['configurations']['cluster-env']['metrics_collector_external_port']
+  else:
+    metric_collector_web_address = default("/configurations/ams-site/timeline.metrics.service.webapp.address", "0.0.0.0:6188")
+    if metric_collector_web_address.find(':') != -1:
+      metric_collector_port = metric_collector_web_address.split(':')[1]
+    else:
+      metric_collector_port = '6188'
+  if default("/configurations/ams-site/timeline.metrics.service.http.policy", "HTTP_ONLY") == "HTTPS_ONLY":
+    metric_collector_protocol = 'https'
+  else:
+    metric_collector_protocol = 'http'
+  pass
+
+# Create current Hadoop Clients  Libs
+stack_version_unformatted = str(config['clusterLevelParams']['stack_version'])
+io_compression_codecs = default("/configurations/core-site/io.compression.codecs", None)
+lzo_enabled = should_install_lzo()
+hadoop_lib_home = stack_root + '/' + stack_version + '/hadoop/lib'
+
+kafka_hosts = default('/clusterHostInfo/kafka_broker_hosts', [])
+has_kafka = len(kafka_hosts) > 0
+
+# Kafka Jaas configs
+kafka_bare_jaas_principal = None
+druid_jaas_file = format('{druid_conf_dir}/druid_jaas.conf')
+if security_enabled and has_kafka and 'kafka_principal_name' in config['configurations']['kafka-env'] :
+    # generate KafkaClient jaas config if kafka is kerberoized
+    _kafka_principal_name = default("/configurations/kafka-env/kafka_principal_name", None)
+    kafka_bare_jaas_principal = get_bare_principal(_kafka_principal_name)
+    _hostname_lowercase = config['agentLevelParams']['hostname'].lower()
+    _druid_principal_name = config['configurations']['druid-common']['druid.escalator.internalClientPrincipal']
+    druid_jaas_principal = _druid_principal_name.replace('_HOST',_hostname_lowercase)
+    druid_keytab_path = config['configurations']['druid-common']['druid.escalator.internalClientKeytab']