A very high volume of messages written to the WebSphere Liberty messages.log has the ability to severely impact the performance of a server.

[fmhwong]

Note: This feature was moved from GHE. Please refer to 33252 for previous design discussion.

Description

It has been found that the writing of a very high volume of messages to the WebSphere Liberty messages.log has the ability to severely impact the performance of a server. In this case, the z/OS Connect product running within a Liberty server was writing millions of GMOMW0005E messages to messages.log. The attempts to write all these messages to a file caused contention for the file, this in turn cause the pthread_yield function to be called. The pthread_yield processing involves setting a STIMER and as STIMER is native z/OS function, zIIP eligible work had wait for the STIMER processing to run on CPs. This had a catastrophic impact on the performance of the system and it was very difficult to identify the root cause.

It has also been reported that files that the messages are written to (trace.log or messages.log) can fill up the zFS if the rate is high enough and the server is up long enough

The existing Liberty function "hideMessage" causes messages to be redirected from messages.log to trace.log, but that does not help in this scenario as it redirects the messages from messages.log to trace.log and the messages are still written to a file.

While the problem was observed with z/OS Connect messages, any message that goes through the Liberty logging/tracing mechanism has the ability to trigger this problem.

We would like a solution which would avoid these performance and file size issues.
A possible solution would be an option which allows any messages reported via the Liberty tracing and logging mechanism to be completely suppressed. A configuration that would allow a number of the messages to be logged until a threashold is reached, so that any failures are not totally silent, but are not allowed to cause the performance problems might be useful.

There could be other routes to a resolution.

---

Documents

When available, add links to required feature documents. Use "N/A" to mark particular documents which are not required by the feature.

• Externally raised requests for enhancements:
  • Aha LIBERTY-I-131: Link to the Aha idea (also add a private comment to link to this issue in the Aha idea)
    • Feature owner adds label Aha idea
  • Open Liberty Feature Request: Link to the OL GH issue
    • Feature owner adds label Requested feature
• Aha: Externally raised RFE (Aha)
  • Link the RFE with this issue, ensure you use a "private" link.
• UFO: https://ibm.box.com/s/k59dxybo26il3sclofejgu3cypi9afwo
• FTS: Link to Feature Test Summary
• Beta Blog: BETA BLOG - Log Throttling #33467

---

Process Overview

• Prioritization
• Design
• Implementation
• Legal and Translation
• GA
  • Focal Point Approvals
• Other Deliverables

General Instructions

The process steps occur roughly in the order as presented. Process steps occasionally overlap.

Each process step has a number of tasks which must be completed or must be marked as not applicable ("N/A").

Unless otherwise indicated, the tasks are the responsibility of the feature owner or a delegate of the feature owner.

If you need assistance, reach out to the OpenLiberty/release-architect.

Important: Labels are used to trigger particular steps and must be added as indicated.

---

Prioritization (Complete Before Development Starts)

The OpenLiberty/chief-architect and area leads are responsible for prioritizing the features and determining which features are being actively worked on.

Prioritization

• Feature owner adds label Prioritization - Requested
  • This puts the feature on the radar of the OpenLiberty/chief-architect and OpenLiberty/project-manager. They are responsible for querying for new features that need to be prioritized.
• OpenLiberty/project-manager adds feature to the "New" column of the Liberty confidential project board
• Priority assigned
  • Attend the Liberty Backlog Prioritization meeting
  • Prioritization - Requested label removed (OpenLiberty/project-manager or feature owner)

---

Design (Complete Before Development Starts)

Design preliminaries determine whether a formal design, which will be provided by an Upcoming Feature Overview (UFO) document, must be created and reviewed. A formal design is required if the feature requires any of the following: UI, Serviceability, SVT, Performance testing, or non-trivial documentation/ID. Furthermore, each identified item places a blocking requirement on another team so it must be identified early in the process. The feature owner may check-off the item if they know it doesn't apply, but otherwise they should work with the focal point to determine what work, if any, will be necessary and make them aware of it.

Design Preliminaries

• UI requirements identified, or N/A. (Feature owner and UI focal point)
• Accessibility requirements identified, or N/A. (Feature owner and Accessibility focal point)
• ID requirements identified, or N/A. (Feature owner and ID focal point)
  • Refer to Documenting Open Liberty.
  • Feature owner adds label ID Required, if non-trivial documentation needs to be created by the ID team.
  • ID adds label ID Required - Trivial, if no design will be performed and only trivial ID updates are needed.
• Serviceability requirements identified, or N/A. (Feature owner and Serviceability focal point)
• SVT requirements identified, or N/A. (Feature owner and SVT focal point)
• Performance testing requirements identified, or N/A. (Feature owner and Performance focal point)

Design

• POC Design / UFO review requested.
  • Feature owner adds label Design Review Request
• POC Design / UFO review scheduled.
  • Follow the Adding an Agenda Item instructions
• POC Design / UFO review completed.
• POC / UFO Review follow-ons completed.
• POC Design / UFO approval requested.
  • Feature owner adds label Design Approval Request
• Design / UFO approved. (OpenLiberty/chief-architect) or N/A
  • (OpenLiberty/chief-architect) adds label Design Approved
  • Add the public link to the UFO in Box to the Documents section.
  • The UFO must always accurately reflect the final implementation of the feature. Any changes must be first approved. Afterwards, update the UFO by creating a copy of the original approved slide(s) at the end of the deck and prepend "OLD" to the title(s). A single updated copy of the slide(s) should take the original's place, and have its title(s) prepended with "UPDATED".

No Design

• No Design requested.
  • Feature owner adds label No Design Approval Request
• No Design / No UFO approved. (OpenLiberty/chief-architect) or N/A
  • Approver adds label No Design Approved
• Feature / Capability stabilization or discontinuation or N/A
  • Feature owner adds label Product Management Approval Request and notifies OpenLiberty/product-management
  • Approver adds label Product Management Approved (OpenLiberty/product-management)

FAT Documentation

• "Feature Test Summary" child task created
  • Use the Feature Test Summary Template
  • Add FTS issue link to the Documents section.

---

Implementation

A feature must be prioritized before any implementation work may begin to be delivered (inaccessible/no-ship). However, a design focused approach should still be applied to features, and developers should think about the feature design prior to writing and delivering any code.
All new Liberty content must be inaccessible in our GA releases until it is Feature Complete by either marking it kind=noship or beta fencing it.
Code may not GA until this feature has obtained the Design Approved or No Design Approved label, along with all other tasks outlined in the GA section.

Feature Development Begins

• Add the In Progress label

Legal and Translation

In order to avoid last minute blockers and significant disruptions to the feature, the legal items need to be done as early in the feature process as possible, either in design or as early into the development as possible. Similarly, translation is to be done concurrently with development. Both MUST be completed before GA is requested.

Innovation (Complete 1 week before Feature Complete Date)

• Consider whether any aspects of the feature may be patentable. If any identified, disclosures have been submitted.

Legal (Complete before Feature Complete Date)

• Changed or new open source libraries are cleared and approved, or N/A. (Legal Release Services/Cass Tucker/Release PM).

Translation (Complete by Feature Complete Date)

• PII (Program Integrated Information) updates are merged (i.e. all English strings due for translation have been delivered), or N/A.

---

GA

A feature is ready to GA after it is Feature Complete and has obtained all necessary Focal Point Approvals.

Feature Complete

• Feature implementation and tests completed.
  • All PRs are merged.
  • All related/child issues are closed.
  • All stop ship issues are completed.
• Legal: all necessary approvals granted.
• Innovation: IP identified and any applicable disclosures submitted
• Translation: Feature may only proceed to GA if it has either Translation - Complete or Translation - Missing label
  • If all translation has been delivered to release branch, feature owner adds label Translation - Complete.
  • If missing translation does not cause a break in functionality, nor a security or production outage risk, feature owner adds label Translation - Missing.
    • Once all missing translations are delivered, the Translation - Missing label is replaced with Translation - Complete.
  • If missing translation could cause a break in functionality or a security or production outage risk, feature owner adds the Translation - Blocked label.
    • Featues with Translation - Blocked may NOT proceed to GA until the label has been replaced with either Translation - Missing or Translation - Complete.
  • For further guidance, contact Globalization focal point or the Release Architect.
• GA development complete and feature ready for inclusion in a GA release
  • Add label target:ga and the appropriate target:YY00X (where YY00X is the targeted GA version).
  • Inclusion in a release requires the completion of all Focal Point Approvals.

Focal Point Approvals (Complete by Feature Complete Date)

These occur only after GA of this feature is requested (by adding a target:ga label). GA of this feature may not occur until all approvals are obtained.

All Features

• APIs/Externals - Externals have been reviewed or N/A. (websphere/externals-approvers)
  • Approver adds label focalApproved:externals
• Demo - Demo is scheduled for an upcoming EOI or N/A. (websphere/demo-approvers
  • Add comment @websphere/demo-approvers Demo scheduled for EOI [Iteration Number] to this issue.
  • Approver adds label focalApproved:demo.
• FAT - All Tests complete and running successfully in SOE or N/A. (websphere/fat-approvers)
  • Approver adds label focalApproved:fat.

Design Approved Features

• ID - Documentation is complete or N/A. (websphere/id-approvers)
  • Approver adds label focalApproved:id.

  • NOTE: If only trivial documentation changes are required, you may reach out to the ID Feature Focal to request a ID Required - Trivial label. Unlike features with regular ID requirement, those with ID Required - Trivial label do not have a hard requirement for a Design/UFO.

• InstantOn - InstantOn capable or N/A. (websphere/instanton-approvers)
  • Approver adds label focalApproved:instantOn.
• Performance - Performance testing is complete or N/A. (websphere/performance-approvers)
  • Approver adds label focalApproved:performance.
• Serviceability - Serviceability has been addressed or N/A. (websphere/serviceability-approvers)
  • Approver adds label focalApproved:sve.
• STE - Skills Transfer Education chart deck is complete or N/A. (websphere/ste-approvers)
  • Approver adds label focalApproved:ste.
• SVT - System Verification Test is complete or N/A. (websphere/svt-approvers)
  • Approver adds label focalApproved:svt.

Post GM (Complete before GA)

• After confirming this feature has been included in the GM driver, feature owner closes this issue.

Post GA

• Remove the target:ga and target:YY00X labels, and add the appropriate release:YY00X. (OpenLiberty/release-manager)

---

Other Deliverables

• Standalone Feature Blog Post - A blog post on IBM Community specifically about your feature or N/A. (Feature Owner and OpenLiberty/release-architect)
  • This should be strongly considered for larger or more prominent features.
• Dev Experience - Developer Experience & Tools work is complete or N/A. (OpenLiberty/dev-experience-assessment)

---