Hercules 4.8 Crash after QETH/OSA "Permission Denied" error

[JimDooey]

Hercules is crashing during startup of z/OS 3.1.

I have run this OS in the past when Hercules version 4.8 was released, however, now I’m getting CPU hung messages. I have tried different NUMCPU and ENGINE settings to no avail.

This execution is using the original DASD images. I’ve deleted my shadow 1 files in attempt to start the system.

No rush is needed for a solution as I’m still customizing earlier z/OS versions.

The GUI screen shows I’m getting errors on the OSA device, but I cannot IPL completely to start a 3270 session and update the TCP/IP profile.

• HerculesCrash.zip

The zip file contains:

1. Hercules.7z (compressed dump file)
2. Hercules Logfile.002.txt (log file entry)
3. zOS310.cnf (Hercules configuration file)
4. zOS310.ini (HercGUI settings file)
5. zOS310.rc (Hercules run commands file)
6. zOS HercGUI.lnk (Shortcut used to start HercGUI and OS)

[Fish-Git]

Introduction:

This execution is using the original DASD images.

I’m getting errors on the OSA device, but I cannot IPL completely to start a 3270 session and update the TCP/IP profile.

2. Hercules Logfile.002.txt (log file entry)

This issue will likely only occur on Windows, and is being caused by the way the Windows CTCI-WIN product is designed: once a virtual adapter/interface is enabled (brought "up") and is running (IFF_UP is set), its settings cannot be changed. In order to modify its settings, you first need to disable (bring "down") the adapter/interface (clear the IFF_UP flag). Then you can change your settings, and bring it back up again afterwards (i.e. re-enable it by setting IFF_UP again).

As I said, this is a limitation of CTCI-WIN.

Because of this, certain specific types of z/OS recovery procedures will ultimately fail, leading to the current situation (CRASH).

There is no fix available at this time. I'm currently looking into it. The fix ultimately involves modifying how Hercules handles RRH_TYPE_ULP --> PUK_TYPE_DISABLE, which ends up calling qeth_disable_interface for Layer 3 tun devices.

We'll eventually figure out the proper Hercules fix for this, but in the mean time, there is fortunately a very simple workaround is available:

---

Workaround:

Simply remove your Hercules device statement for your QETH/OSA device, until such time as you can manage to properly update your z/OS TCP/IP profile. This will prevent z/OS from encountering its error situation, thereby preventing it from attempting its recovery and ultimately the Hercules CRASH.

This will allow z/OS to finish coming all the way up, thus allowing you to logon to TSO and fix your TCP/IP profile, Once your TCP/IP profile has been fixed, do a clean shutdown of z/OS, and POWER OFF (i.e. exit) Hercules, add your desired Hercules QETH/OSA device statement to your Hercules configuration file, and bring up z/OS again. THIS time it should work okay (presuming you updated your TCPIP profile correctly of course!).

---

Analysis:

What is happening is this: because your Hercules OSA device didn't match what z/OS was expecting, some type of unusual error occurred, causing z/OS to perform recovery actions. The recovery actions that z/OS performed, caused it to send a RRH_TYPE_ULP --> PUK_TYPE_DISABLE, which disabled the OSA device.

Then, it tried (presumably!) sending a (packet? command?) to either it's "data" device or to its "control" device, causing Hercules is issue a "write packet" to that corresponding device, which ultimate FAILED with a "Permission Denied" error due to the fact that the adapter/interface was still disabled!

(Excerpt from user's Herc log file):

08:39:24.101 0x00000AF0 HHC03800I 0:1501 OSA: Adapter mode set to Layer 3
08:39:24.144 0x00000AF0 HHC03997I 0:1501 OSA: tun0: using MAC address 02:00:5e:a3:be:84
08:39:24.144 0x00000AF0 HHC03997I 0:1501 OSA: tun0: using IP address 192.168.1.31
08:39:24.144 0x00000AF0 HHC03997I 0:1501 OSA: tun0: using subnet mask 255.255.255.0
08:39:24.144 0x00000AF0 HHC03997I 0:1501 OSA: tun0: using MTU 1500
08:39:24.144 0x00000AF0 HHC03997I 0:1501 OSA: tun0: using drive MAC address 96:7a:59:e5:d2:bf
08:39:24.144 0x00000AF0 HHC03997I 0:1501 OSA: tun0: using drive IP address fe80::967a:59ff:fee5:d2bf
08:39:24.150 0x00000AF0 HHC03805I 0:1501 OSA: tun0: Register guest IP address 192.168.100.150


08:39:44.413 0x00000AF0 HHC03997I 0:1501 OSA: tun0: not using MAC address 02:00:5e:a3:be:84
08:39:44.413 0x00000AF0 HHC03997I 0:1501 OSA: tun0: not using IP address 192.168.1.31
08:39:44.413 0x00000AF0 HHC03997I 0:1501 OSA: tun0: not using subnet mask 255.255.255.0
08:39:44.413 0x00000AF0 HHC03997I 0:1501 OSA: tun0: not using MTU 1500
08:39:44.417 0x00000AF0 HHC00905I 0:1502 OSA: HSCH recognized for data device


08:39:44.448 0x00000AF0 HHC00911E 0:1502 OSA: Error writing to device tun0: 13 Permission denied
08:39:44.448 0x00000AF0 HHC00007I Previous message from function 'write_packet' at qeth.c(2859)


08:40:26.961 0x00000AF0 HHC00822S PROCESSOR CP00 APPEARS TO BE HUNG!

Recall that, as previously explained, CTCI-WIN will not allow you to read or write to/from (will not let you use) your emulated adapter/interface while it is "down" (i.e. disabled by virtue of the fact that the IFF_UP flag is still off).

As a result of the "Permission Denied" error, Hercules's intlock was still held by the device thread (which was stuck!) thereby bringing the entire system to a screeching halt leading to the CRASH (all CPUs being hung waiting for intlock.

---

I'm going to add the "Ongoing" label to this issue since the ultimate fix is FAR from being simple or straightforward, and a lot of collaborative effort is going to be involved, which is going to take some time.

It's not a priority however since a simple workaround is available. But we'll get it fixed eventually I'm sure.

Hope that helps!