Add support for ICSF hardware keys in API ML

docs/appendix/zowe-yaml-configuration.md

@@ -147,15 +147,15 @@ In the `zowe.yaml`, you can define default values which can be overridden in mor
 In Zowe YAML configuration, the certificate definition shares the same format which can be used in several configuration entries. For example, `zowe.certificate`, `components.<component>.certificate`, and `haInstances.<ha-instance>.components.<component>.certificate`. The certificate definition may include the following entries:
 
 - **keystore.type**  
- Specifies the type of the keystore. If you are using keystore, this value usually should be `PKCS12`. If you are using keyring, this value should be `JCERACFKS`.
+ Specifies the type of the keystore. If you are using keystore, this value usually should be `PKCS12`. If you are using z/OS keyring, this value should be one of `JCEHYBRIDRACFKS`, `JCECCARACFKS`, `JCERACFKS`.
 - **keystore.file**  
  Specifies the path of the keystore file. If you are using keyring, this should look like `safkeyring://<keyring-owner>/<keyring-name>`. For example, `safkeyring://ZWESVUSR/ZoweKeyring`.
 - **keystore.password**  
  Specifies the password of the keystore.
 - **keystore.alias**  
  Represents the alias name of the certificate stored in keystore. If you are using keyring, this is the certificate label connected to the keyring.
 - **truststore.type**  
- Specifies the type of the truststore file. If you are using keystore, this value usually should be `PKCS12`. If you are using keyring, this value should be `JCERACFKS`.
+ Specifies the type of the truststore file. If you are using keystore, this value usually should be `PKCS12`. If you are using z/OS keyring, this value should be one of `JCEHYBRIDRACFKS`, `JCECCARACFKS`, `JCERACFKS`.
 - **truststore.file**  
  Specifies the path to the truststore file. If you are using keyring, this should look like `safkeyring://<keyring-owner>/<keyring-name>`, and usually will be the same value of `keystore.file`.
 - **truststore.password**  
@@ -194,13 +194,14 @@ The high-level configuration `zowe` supports these definitions:
 - **zowe.externalDomains**  
  Specifies a list of external domains to be used by the Zowe instance. This configuration is an array of domain name strings.
  In Sysplex deployment, this value is the DVIPA domain name defined in Sysplex Distributor. For example,
-   
+
    ```yaml
    zowe:
     externalDomains:
     - external.my-company.com
     - additional-dvipa-domain.my-company.com
    ```
+
  In Kubernetes deployment, this value is the domain name you will use to access your Zowe running in a Kubernetes cluster.
 - **zowe.externalPort**  
  Specifies the port that is to be exposed to external Zowe users. By default, this value is set based on Zowe APIML Gateway port.
@@ -218,7 +219,7 @@ The high-level configuration `zowe` supports these definitions:
     environments:
       MY_NEW_ENV: value-of-my-env
    ```
- 
+
  :::note
  Variables defined here are global to all Zowe components, on all HA instances.
 

docs/extend/extend-apiml/onboard-plain-java-enabler.md

@@ -727,7 +727,7 @@ The following example shows enabler configuration with keyrings.
 
 **Example:**
 
-```
+```yaml
 ssl:
     keyAlias: localhost
     keyPassword: password

docs/troubleshoot/troubleshoot-zos-certificate.md

@@ -19,10 +19,11 @@ As an API Mediation Layer user, you may encounter problems when configuring cert
 **Symptoms**
 
 Some Zowe Desktop applications do not work when Zowe creates a PKCS12 keystore. A message may appear in the log such as the following:
+
 * ZWES1060W Failed to init TLS environment, rc=1(Handle is not valid)
 * ZWES1065E Failed to configure https server. Check agent https setting.
 
-These messages indicate that ZSS cannot read the generated keystore. As such, parts of Zowe are not functional. 
+These messages indicate that ZSS cannot read the generated keystore. As such, parts of Zowe are not functional.
 
 **Solutions**
 
@@ -68,7 +69,8 @@ The certificate appears to be correct, but the Gateway and the Discovery Service
 The password is only used for USS PKCS12 certificate files. The keyring is protected by SAF permissions. Note that in some configurations, Zowe does not work if the password value is empty in the keyring configuration. We recommended that you assign a value to `password` as shown in the following example:
 
 **Example:**
-```
+
+```yaml
 certificate:
     keystore:
       type: JCERACFKS
@@ -106,9 +108,9 @@ You used an external certificate and Single Sign-On to deploy Zowe. When you log
 **Solution:**
 
 This issue might occur when you use a Zowe version of 1.12.0 or later. To resolve the issue, you can download your external root certificate and intermediate certificates in PEM format. Then, add the following parameter in the `zowe.yaml` file.
- 
+
 ```environments.ZWED_node_https_certificateAuthorities: "/path/to/zowe/keystore/local_ca/localca.cer-ebcdic","/path/to/carootcert.pem","/path/to/caintermediatecert.pem"```
- 
+
 Recycle your Zowe server. You should be able to log in to the Zowe Desktop successfully now.
 
 ## Browser unable to connect due to a CIPHER error
@@ -145,7 +147,7 @@ To do this, first locate the `$JAVA_HOME/lib/security/java.security` file. You c
 
 - Method 2: By inspecting the `STDOUT` JES spool file for the `ZWESLSTC` started task that launches the API Mediation Layer.
 
-   
+
 In the `java.security` file, there is a parameter value for `jdk.tls.disabledAlgorithms`.
 
 **Example:**
@@ -177,21 +179,21 @@ however they are unable to communicate with each other.
 
 Externally, the status of the API Gateway homepage displays **!** icons against the API Catalog, Discovery Service and Authentication Service (shown on the left side image below)
  which do not progress to green tick icons as normally occurs during successful startup (shown on the right side image below).
- 
+
 <img src={require("../images/api-mediation/apiml-startup.png").default} alt="Zowe API Mediation Layer Startup" width="600px"/> 
 
 The Zowe desktop is able to start but logon fails.
- 
+
 The log contains messages to indicate that connections are being reset. For example, the following message shows that the API Gateway `ZWEAG` is unable to connect to the API Discovery service, by default 7553.
 
-``` 
+```log
 <ZWEAGW1:DiscoveryClient-InstanceInfoReplicator-0:16843005> ZWESVUSR INFO  (o.a.h.i.c.DefaultHttpClient) I/O exception (java.net.SocketException) caught when connecting to {s}->https://<host>:<disovery_server_port>: Connection reset
 2021-01-26 15:21:43.302 <ZWEAGW1:DiscoveryClient-InstanceInfoReplicator-0:16843005> ZWESVUSR DEBUG (o.a.h.i.c.DefaultHttpClient) Connection reset
 java.net.SocketException: Connection reset
 ```
 
 The Zowe desktop is able to be displayed in a browser but fails to logon.
- 
+
 **Solution:**
 
 Check that the Zowe certificate has been configured as a client certificate, and not just as a server certificate. For more informtion, see More detail can be found in [Configuring certificates overview](../user-guide/configure-certificates).
@@ -203,7 +205,8 @@ Check that the Zowe certificate has been configured as a client certificate, and
 Java z/OS components of Zowe are unable to read certificates from a keyring. This problem may appear as an error as in the following example where Java treats the SAF keyring as a file.
 
 **Example:**
-```
+
+```log
 Caused by: java.io.FileNotFoundException: safkeyring:/ZWESVUSR/ZoweKeyring
 at java.io.FileInputStream.open(FileInputStream.java:212)
 at java.io.FileInputStream.<init>(FileInputStream.java:152)
@@ -245,10 +248,8 @@ Make sure that the private key stored in the keyring is not encrypted by a passw
 
 If you see one or more of the following messages in the logs, the cause is keyring configuration.
 
-
 - ZWED0148E - Exception thrown when reading SAF keyring, e= Error: R_datalib call failed: function code: 01, SAF rc: `number`, RACF rc: `number`, RACF rsn: `number`
 
-
 * java.io.IOException: R_datalib (IRRSDL00) error: profile for ring not found (`number`, `number`, `number`)
 
 You may also see the following log message:
@@ -265,7 +266,7 @@ You may also see the following log message:
 **Example:** 
  If ZWED0148E contains the following message, it indicates that Zowe's local certificate authority (local CA) `ZoweCert`, the certificate `jwtsecret`, or the Zowe certificate `localhost` does not exist in the Zowe keyring. 
 
-```
+```log
 2021-01-18 10:16:33.601 <ZWED:16847011> ZWESVUSR WARN (_zsf.bootstrap,webserver.js:156) ZWED0148E - Exception thrown when reading SAF keyring, e= TypeError: R_datalib call failed: function code: 01, SAF rc: 8, RACF rc: 8, RACF rsn: 44
 at Object.getPemEncodedData (/software/zowev15/1.15.0/components/app-server/share/zlux-server-framework/node_modules/keyring_js/index.js:21:26)
 ```
@@ -276,7 +277,7 @@ If you are using your own trusted CA certificate in the keyring, and the name is
 
 If you are using Zowe's local CA certificate and you still receive **ZWED0148E**, you may find the following message in the same log.
 
-```
+```json
   "https": {
     "ipAddresses": [
       "0.0.0.0"
@@ -334,6 +335,7 @@ API ML components do not start properly because they fail to load the JCERACFKS
 The keyring, however, is configured correctly and the STC user can access it. 
 
 **Examples:**
+
 ```
 2023-06-27 13:07:45.138 ..35m<ZWEACS1:main:67502789>..0;39m APIMTST ..31mERROR..0;39m ..36m(o.a.t.u.n.SSLUtilBase)..0;39m Failed to load keystore type .JCERACFKS. with path .safkeyring://ZWESVUSR/ZOWERING. due to .JCERACFKS not found.
 java.security.KeyStoreException: JCERACFKS not found                                                                                                            
@@ -352,7 +354,8 @@ JCERACFKS KeyStore not available
 
 In Java 11 releases before 11.0.17.0, the `IBMZSecurity` security provider is not enabled by default. Locate the `java.security` configuration file in the `$JAVA_HOME/conf/security` USS directory 
 and open the file for editing. Modify the list of security providers and insert `IBMZSecurity` on second position. The list of enabled security providers should resemble the following series:
-```
+
+```properties
 security.provider.1=OpenJCEPlus
 security.provider.2=IBMZSecurity
 security.provider.3=SUN
@@ -368,4 +371,69 @@ security.provider.12=JdkLDAP
 security.provider.13=JdkSASL
 security.provider.14=SunPKCS11
 ```
+
 For more information see the steps in [Enabling the IBMZSecurity provider](https://www.ibm.com/docs/en/semeru-runtime-ce-z/11?topic=guide-ibmzsecurity#ibmzsecurity__enabling_z_provider__title__1).
+
+## Failed to load JCECCARACFKS keyring when using ICSF under ACF2
+
+The issue below may occur only under ACF2, due to differences in how ACF2 handles digital certificate
+and key resource access compared to TSS and RACF.
+
+**Symptom**
+
+The Zowe log contains the following ERROR message:  
+`com.ibm.crypto.ibmjcehybrid.provider.IBMJCEHybridException: Failover exhausted, all registered providers attempted and failed.`
+
+**Explanation**
+
+This error occurs when the `Java Cryptography Extension` (JCE) provider cannot access the `RSA` key dataset defined for Zowe.
+In ACF2, access to RSA key resources is controlled through the `CSFKEYS` resource class,
+which governs permissions to use or read private keys stored in the security database.
+If the Zowe started task ID (`ZWESVUSR`) lacks the appropriate `READ` access to the RSA key resource, all configured JCE providers
+fail to initialize, resulting in this failover error.
+
+**Solution**
+
+Grant the Zowe started task user ID `READ` access to the RSA key stored in PKDS dataset and rebuild the `CSFKEYS` class:
+
+**Example:**
+
+```
+SET RESOURCE(CSK)
+RECKEY ZOWERSAKEY ADD(USER(ZWESVUSR) SERVICE(READ) ALLOW)
+F ACF2,REBUILD(CSK)
+```
+
+Where:
+* `ZOWERSAKEY` - is the PKDS label for the RSA key linked to the certificate in the Zowe keyring.
+* `ZWESVUSR` - is the Zowe started task user ID.
+
+**Symptom**
+
+The Zowe log contains the following error message:  
+`IBMJCEHybridException: Errors encountered loading keyring. Keyring could not be loaded as a JCECCARACFKS or JCERACFKS keystore.
+Exception#0 java.io.IOException: R_datalib (IRRSDL00) error: not RACF authorized to use the requested service (8, 8, 8)`
+
+**Explanation**
+
+This error indicates that the Zowe started task user ID (`ZWESVUSR`) does not have the required authorization to
+access the R_datalib callable service (`IRRSDL00`).
+In ACF2, this corresponds to the `FACILITY` class resource `DIGTCERT.LISTRING`.
+Without this access, Zowe cannot load the required keyring for establishing secure TLS connections.
+
+**Solution**
+
+Grant the Zowe started task user ID the necessary `READ` access to the `IRR.DIGTCERT.LISTRING` resource and rebuild the `FACILITY` class:
+
+ **Important:** It should be USER/UID, but not ROLE to whom access is given.
+
+**Example:**
+
+```
+SET RESOURCE(FAC)
+RECKEY IRR ADD(DIGTCERT.LISTRING USER(ZWESVUSR) SERVICE(READ) ALLOW)
+F ACF2,REBUILD(FAC)
+```
+
+Where:
+* `ZWESVUSR` - is the Zowe started task user ID.

docs/user-guide/api-mediation-icsf-keyring.md

@@ -0,0 +1,70 @@
+# Zowe API Mediation Layer - Using ICSF Hardware Private Key
+
+Zowe version 3.5.0 introduces support in the API Mediation Layer for ICSF-backed private keys.
+Previously this was supported only via [AT-TLS](../../user-guide/configuring-at-tls-for-zowe-server.md) with limitations whereas z/OSMF was required as the selected authentication provider and Personal Access Tokens could not be used.
+
+## Configuring the z/OS system
+
+Enabling Zowe API Mediation Layer to use ICSF keyrings requires changes to the server user authorization and Java security policy.
+
+### Server user permissions
+
+The Zowe server user must be granted access to specific `CSFSERV` class resources in order to interact with ICSF.
+Ensure that the user has `READ` access to the following resources in the `CSFSERV` class:
+
+Resource|Description
+---|---
+CSFIQF|ICSF Query Facility callable service
+CSFOWH|one-way hash generate callable service
+CSFRNG|random number generate callable service
+CSFRNGL|random number generate long callable service
+CSFPKG|PKA key generate callable service
+CSFDSG|digital signature generate service
+CSFDSV|digital signature verify callable service
+CSFPKX|PKA Public Key Extract callable service
+CSFPKI|PKA key import callable service
+CSFEDH|ECC Diffie-Hellman callable service
+
+These permissions are necessary for key generation, encryption/decryption, signing of JWT tokens and other cryptographic operations performed via ICSF.
+
+### Java configuration
+
+:::tip
+
+Zowe bundles an updated version of the java security policy file. This can be enabled by setting:
+
+```yaml
+zowe:
+  environments:
+    JVM_SECURITY_PROPERTIES_OVERRIDE: true
+```
+
+Note this will override the JVM-defined cryptography provider list.
+
+:::
+
+Using ICSF hardware keys in API ML requires changes to the Java security configuration.
+Perform the following changes in the `java.security` file, typically located in `$JAVA_HOME/conf/security` directory:
+
+1. Ensure the following cryptography providers are installed at the top of the list:
+
+```plaintext
+security.provider.1=IBMJCEHYBRID
+security.provider.2=IBMJCECCA
+```
+
+For more information, refer to the `IBM Semeru Runtime Certified Edition for z/OS` IBM product documentation, articles `Installing security providers`, `IBMJCECCA` and `IBMJCEHYBRID`.
+
+## Configuring Zowe to Use ICSF Keyrings
+
+Update the `zowe.certificate` section in your `zowe.yaml` configuration file as follows:
+
+* Set `zowe.certificate.keystore.type` to `JCEHYBRIDRACFKS`
+
+* Set `zowe.certificate.truststore.type` to `JCEHYBRIDRACFKS`
+
+Make sure `zowe.certificate.trustore.file` and `zowe.certificate.keystore.file` has protocol `safkeyring://` or `safkeyringhybridjce://`
+
+## Troubleshooting
+
+For more information about troubleshooting, check [troubleshooting](../../docs/troubleshoot/troubleshoot-zos-certificate.md).