Some questions when use hue connect to Yarn cluster with kerberos

[Meepoljdx]

• Software version
  • HUE: 4.11.0
  • HDP: 3.1.1.5
  • OS: Centos7

I use hue connect to a yarn cluster with kerberos security, i can get the yarn application, but when i want to get some task log (e.g stderr or stdout) the brower return message:

Failed to retrieve executor log: HTTPSConnectionPool(host='hdp001.datasw', port=8044): Max retries exceeded with url: /node/containerlogs/container_e204_1688606157391_34220_01_000001/hdfs/stdout?doAs=admin&start=-1000000 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

Failed to retrieve executor log: HTTPSConnectionPool(host='hdp003.datasw', port=8044): Max retries exceeded with url: /node/containerlogs/container_e205_1690275359367_0028_01_000001/hdfs/stderr?doAs=admin&start=-1000000 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))

When i look hue log, i found those output:

[25/Jul/2023 17:11:40 +0800] spark_history_server_api DEBUG    Getting real spark app id application_1690275359367_0028/1 for Spark job application_1690275359367_0028
/home/workspace/hue-release-4.11.0/build/env/lib/python3.9/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host 'hdp002.datasw'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
[25/Jul/2023 17:11:40 +0800] resource     DEBUG    GET <class 'requests_kerberos.kerberos_.HTTPKerberosAuth'> http://hdp002.datasw:18081/api/v1/applications/application_1690275359367_0028/1/executors  returned in 26ms 200 2850 [ {
  "id" : "driver",
  "hostPort" : "hdp003.datasw:40508",
  "isActive" : true,
  "rddBlocks" : 0,
  "memoryUsed" : 0,
  "diskUsed" : 0,
  "totalCores" : 0,
  "maxTasks" : 0,
  "activeTasks" : 0,
  "failedTasks" : 0,
  "completedTasks" : 0,
  "totalTasks" : 0,
  "totalDuration" : 0,
  "totalGCTime" : 0,
  "totalInputBytes" : 0,
  "totalShuffleRead" : 0,
  "totalShuffleWrite" : 0,
  "isBlacklisted" : false,
  "maxMemory" : 97832140,
  "addTime" : "2023-07-25T09:10:14.165GMT",
  "executorLogs" : {
    "stdout" : "https://hdp003.datasw:8044/node/containerlogs/container_e205_1690275359367_0028_01_000001/hdfs/stdout?start=-4096",
    "stderr" : "https://hdp003.datasw:8044/node/containerlogs/container_e205_1690275359367_0028_01_000001/hdfs/stderr?start=-4096"
  },
  "memoryMetrics" : {
    "usedOnHeapStorageMemory" : 0,
    "usedOffHeapStorageMemory" : 0,
    "totalOnHeapStorageMemory" : 97832140,
    "totalOffHeapStorageMemory" : 0
  }
}, {
  "id" : "2",
  "hostPort" : "hdp005.datasw:34542",
  "isActive" : true,
  "rddBlocks" : 0,
  "memoryUsed" : 0,
  "diskUsed" : 0,
  "totalCores" : 2,
  "maxTasks" : 2,
  "activeTasks" : 0,
  "failedTasks" : 0,
  "completedTasks" : 3,
  "totalTasks" : 3,
  "totalDuration" : 3577,
  "totalGCTime" : 179,
  "totalInputBytes" : 0,
  "totalShuffleRead" : 0,
  "totalShuffleWrite" : 0,
  "isBlacklisted" : false,
  "maxMemory" : 956615884,
  "addTime" : "2023-07-25T09:10:20.816GMT",
  "executorLogs" : {
    "stdout" : "https://hdp005.datasw:8044/node/containerlogs/container_e205_1690275359367_0028_01_000003/hdfs/stdout?start=-4096",
    "stderr" : "https://hdp005.datasw:8044/node/containerlogs/container_e205_1690275359367_0028_01_000003/hdfs/stderr?start=-4096"
  },
  "memoryMetrics" : {
    "usedOnHeapStorageMemory" : 0,
    "usedOffHeapStorageMemory" : 0,
    "totalOnHeapStorageMemory" : 956615884,
    "totalOffHeapStorageMemory" : 0
  }
}, {
  "id" : "1",
  "hostPort" : "hdp003.datasw:45295",
  "isActive" : true,
  "rddBlocks" : 0,
  "mem...
[25/Jul/2023 17:11:40 +0800] http_client  DEBUG    Setting request Session
[25/Jul/2023 17:11:40 +0800] http_client  DEBUG    Setting session adapter for https://hdp003.datasw:8044
/home/workspace/hue-release-4.11.0/build/env/lib/python3.9/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host 'hdp001.datasw'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
[25/Jul/2023 17:11:40 +0800] resource     DEBUG    GET <class 'requests_kerberos.kerberos_.HTTPKerberosAuth'> https://hdp001.datasw:8090/ws/v1/cluster/info  returned in 5ms 200 710 {"clusterInfo":{"id":1690275771003,"startedOn":1690275771003,"state":"STARTED","haState":"STANDBY","rmStateStoreName":"org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore","resourceManagerVersion":"3.1.1.3.1.5.0-152","resourceManagerBuildVersion":"3.1.1.3.1.5.0-152 from e235cd6e7bfe0fe31ee6448b612862c53b45dda9 by jenkins source checksum 9f43f62aaf9a11327a7776a73e196f10","resourceManagerVersionBuiltOn":"2019-12-12T19:47Z","hadoopVersion":"3.1.1.3.1.5.0-152","hadoopBuildVersion":"3.1.1.3.1.5.0-152 from e235cd6e7bfe0fe31ee6448b612862c53b45dda9 by jenkins source checksum 273fb5b93b851c12cbc42c54de1a6c1","hadoopVersionBuiltOn":"2019-12-12T19:46Z","haZooKeeperConnectionState":"CONNECTED"}}

I think this error is caused by the communication between hue and nodemanager, but I don't know how to solve it

This is my hue.ini yarn block:

  [[yarn_clusters]]

    [[[default]]]
      resourcemanager_host=hdp001.datasw
      resourcemanager_port=8050
      submit_to=True
      logical_name=rm1
      security_enabled=true

      # URL of the ResourceManager API
      resourcemanager_api_url=https://hdp001.datasw:8090
      spark_history_server_url=http://hdp002.datasw:18081
      spark_history_server_security_enabled=true
      ssl_cert_ca_verify=False


    [[[ha]]]
      logical_name=rm2
      submit_to=True
      resourcemanager_api_url=https://hdp002.datasw:8090

This is my jobbrower web page:

[Harshg999]

Hey @Meepoljdx, it looks like a SSL error and Hue is not able to talk to HTTPSConnectionPool(host='hdp003.datasw', port=8044).

Can you try setting ssl_cert_ca_verify=False to True and try again?

[Meepoljdx]

It's no use,if i set ssl_cert_ca_verify=False I can get the application list, but can not get the task stderr or stdout log(So I'm not sure if it was hue's problem when calling nodemanger's http request to get the logs). When i set ssl_cert_ca_verify=True, I can't even get application list.