New feature in devc handled out of order

[turkeylurkey]

After adding a new Java file the endpoint is not visible. Tested on Linux.

First start with a kernel image: kernel-slim-java8-openj9-ubi.

Using the demo-devmode project add a command to Dockerfile to copy generate-fearures.xml to the image:
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/generated-features.xml /config/configDropins/overrides/

With dev mode running add the file SystemLivenessCheck.java to the project.
As seen in the output below source compilation is successful.

[INFO] Source compilation was successful.
[INFO] Running liberty:generate-features
[INFO] [AUDIT   ] CWWKT0017I: Web application removed (default_host): http://34038634b045:9080/
[INFO] [AUDIT   ] CWWKZ0009I: The application demo-devmode-maven has stopped successfully.
[INFO] Generated the following features: [cdi-2.0, mpHealth-3.1, mpMetrics-3.0, jaxrs-2.1]
[INFO] Copied file: /home/ibmadmin/demo-devmode/src/main/liberty/config/configDropins/overrides/generated-features.xml to: /tmp/tempConfig12837632527231767519/configDropins/overrides/generated-features.xml
[INFO] Copied file: /home/ibmadmin/demo-devmode/src/main/liberty/config/configDropins/overrides/generated-features.xml to: /tmp/tempConfig12837632527231767519/configDropins/overrides/generated-features.xml
[INFO] Configuration features have been added: [mphealth-3.1]
[INFO] Running liberty:install-feature
[INFO] CWWKM2102I: Using artifact based assembly archive : io.openliberty:openliberty-kernel:null:22.0.0.4:zip.
[INFO] CWWKM2102I: Using installDirectory : /home/ibmadmin/demo-devmode/target/liberty/wlp.
[INFO] CWWKM2102I: Using serverName : defaultServer.
[INFO] CWWKM2102I: Using serverDirectory : /home/ibmadmin/demo-devmode/target/liberty/wlp/usr/servers/defaultServer.
[INFO] Parsing the server file for features and includes: tempConfig12837632527231767519/configDropins/defaults/install_apps_configuration_1491924271.xml
[INFO] Parsing the server file for features and includes: tempConfig12837632527231767519/server.xml
[INFO] Parsing the server file for features and includes: tempConfig12837632527231767519/configDropins/overrides/generated-features.xml
[INFO] Parsing the server file for features and includes: tempConfig12837632527231767519/configDropins/overrides/liberty-plugin-variable-config.xml

A new feature is generated and added to the container.

[INFO] Installing features [mphealth-3.1, mpmetrics-3.0, cdi-2.0, jaxrs-2.1] on container liberty-dev
[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://34038634b045:9080/
[INFO] [AUDIT   ] CWWKZ0003I: The application demo-devmode-maven updated in 1.798 seconds.
[INFO] Copied file: /home/ibmadmin/demo-devmode/src/main/liberty/config/configDropins/overrides/generated-features.xml to: /home/ibmadmin/demo-devmode/target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/generated-features.xml
[INFO] [WARNING ] SRVE0190E: File not found: /health

The web application is started before generated-features.xml is copied to target.
The new endpoint is not found.
If you restart the container the events happen in order and the endpoint is found.
After container restart:

[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://4fb16a83aada:9080/metrics/
[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://4fb16a83aada:9080/ibm/api/
[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://4fb16a83aada:9080/health/
[INFO] [AUDIT   ] CWWKT0016I: Web application available (default_host): http://4fb16a83aada:9080/
[INFO] [AUDIT   ] CWWKZ0001I: Application demo-devmode-maven started in 5.069 seconds.

[TrevCraw]

Attempt to reproduce and see how broken dev mode is when this happens.
Also try to reproduce on another platform. Try a few times to see if intermittent. Try with large feature list to see if slow install features is more likely to cause it.

[kathrynkodama]

EDIT: Based on #1070 (comment) this looks to be a unique issue that is occurring on slower machines where the application is updating before the features are installed. Following the feature definition not found error noted above, if a user makes subsequent source file changes that does not guarantee that the application works as expected as the subsequent file changes may not trigger install feature. If generate-features detects a new feature and install feature is triggered, that may correct the issue. "R" + Enter to restart the container does ensure that the correct features are installed and the application works as expected.

Looks to be a duplicate of #1070 but the suggested workaround of copying server.xml & generated-features.xml from the target dir noted here: #1070 (comment) does not seem to work.

Was not able to reproduce reliably with the current Dockefile on MacOS, Java 11, Maven 3.8.4.

[kathrynkodama]

After further investigation there are two main issues:

1. On Linux when copying single files in a Dockerfile, the single file bound mounts may be corrupt. This inconsistently occurs on Linux.

COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/generated-features.xml /config/configDropins/overrides/

In this scenario, when the generated-features.xml file in the local target directory is updated, the generated-features.xml file on the container is not updated. This is confirmed by checking the inode numbers (ls -i generated-features.xml). Prior to a change the inode numbers are the same, after a change they vary with the file on the container still referencing the old inode number.
This occurs for many of the files on Linux that use a single file mount. The workaround is to copy entire directories in the Dockerfile, ie.

COPY --chown=1001:0  target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides /config/configDropins/overrides

or to restart the server and container with "R" + Enter.

In dev mode, the proposed solution is to restart the container on Linux when a server configuration file has changed. We can extend this existing fix for a similar issue: OpenLiberty/ci.common#171.

2. After restarting the container without rebuilding the image, certain features that were previously installed may be missing. This occurs when using a Liberty Docker kernel image. To install features to the container, dev mode calls liberty:install-feature which in turn calls featureUtility to run on the container.
   When we restart the container without rebuilding the image, we are stopping the container and starting a new container. Since we are not rebuilding the image, the RUN features.sh of the Dockerfile does not run and we are not running liberty:install-feature, so the features will not be properly installed on the new container. This can be confirmed by comparing the features .mf files located on the container (/lib/features/) before restarting and after.

A proposed solution would be to call install features in the restartServer logic in the event that we do not rebuild the image (as RUN features.sh would not have ran): https://github.com/OpenLiberty/ci.common/blob/9ad1bb92bafd1fa3a31bbea52742e176ac7639a2/src/main/java/io/openliberty/tools/common/plugins/util/DevUtil.java#L1820-L1824

Dockerfile for reference:

FROM icr.io/appcafe/open-liberty:kernel-slim-java8-openj9-ubi

# Add config
COPY --chown=1001:0  target/liberty/wlp/usr/servers/defaultServer/server.xml /config/
COPY --chown=1001:0  target/liberty/wlp/usr/servers/defaultServer/bootstrap.properties /config/
COPY --chown=1001:0  target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/liberty-plugin-variable-config.xml /config/configDropins/overrides/
COPY --chown=1001:0 target/liberty/wlp/usr/servers/defaultServer/configDropins/overrides/generated-features.xml /config/configDropins/overrides/

RUN features.sh

# Add application
COPY --chown=1001:0  target/liberty/wlp/usr/servers/defaultServer/apps/demo-devmode-maven.war /config/apps/