DefaultServlet: cached `JarResource` serves a closed `JarFile` → `IllegalStateException: zip file closed` for large (>512 KB) webjar assets after idle

[mafaul]

Environment

• GlassFish 8.0.x, JDK 21, Linux
• A WAR with a webjar on its classpath (e.g. org.webjars:swagger-ui) served directly from the JAR (i.e. not unpacked into the
  docroot), containing a static asset larger than ~512 KB (swagger-ui-bundle.js ≈ 1.4 MB)

Summary

A static resource served by the DefaultServlet from inside a WEB-INF/lib JAR (META-INF/resources/…) returns HTTP 500 with
java.lang.IllegalStateException: zip file closed after the application has been idle. Only resources above the resource-cache
per-object size limit (~512 KB) are affected; smaller siblings from the same JAR keep working. The root cause is a lifecycle mismatch: the
resource cache pins a JarResource that references a JarFile owned by the WebappClassLoader, the classloader closes that handle after
an idle timeout, and cache revalidation reuses the stale JarResource without reopening.

Observed behavior (smoking gun)

• Immediately after deploy: the large asset serves 200 OK.
• Next morning, after the app sat idle overnight with no redeployment: the first request to the same asset returns 500 zip file closed. Smaller assets from the same JAR still return 200.

Steps to reproduce

1. Deploy a WAR that serves a webjar from the JAR with a static file >512 KB (e.g. swagger-ui's swagger-ui-bundle.js).
2. Request the asset once → 200 (it gets cached as a JarResource while the JAR handle is open).
3. Leave the app idle for >90 s (no requests that load classes/resources from those JARs; no redeploy).
4. Request the large asset again → 500 zip file closed. Sub-512 KB assets from the same JAR still serve.

Reproducer: minimal WAR attached (https://github.com/user-attachments/files/28826112/glassfish-webjar-bug-repro.zip) — mvn clean package, deploy webjar-bug.war to GlassFish 8.0.x,
then GET http://localhost:8080/webjar-bug/webjars/swagger-ui/5.31.0/swagger-ui-bundle.js (200), leave the app idle >90 s, GET it again → 500 zip file closed. See
README for the no-wait variant.

Note: verify with curl, not a browser reload — once the asset has loaded 200 the browser serves its cached copy (or gets a 304 Not Modified) and never re-exercises the JAR-stream path. To force a server hit, use curl, DevTools → Network → Disable cache, or append
a cache-buster (…/swagger-ui-bundle.js?cb=1) — the query string is ignored for resource resolution so it still hits the bug.

Stack trace (core frames; servlet filter chain elided)

java.lang.IllegalStateException: zip file closed
    at java.base/java.util.zip.ZipFile.ensureOpen(ZipFile.java:846)
    at java.base/java.util.zip.ZipFile.getInputStream(ZipFile.java:375)
    at java.base/java.util.jar.JarFile.getInputStream(JarFile.java:856)
    at org.apache.naming.resources.WebDirContext$JarResource.streamContent(WebDirContext.java:474)
    at org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:1915)
    at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:911)
    at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:444)

Root cause (class/method names below are stable across the GlassFish/Catalina lineage)

1. The JAR handle is owned by WebappClassLoader (org.glassfish.web.loader.WebappClassLoader, which implements
   JarFileResourcesProvider). It closes its JARs after 90 s of inactivity and self-heals on next access:

   // closeJARs(boolean force)
   if (force || (System.currentTimeMillis() > (lastJarAccessed + 90000))) {
       ... jarFiles[i].close(); jarFiles[i] = null; ...
   }
   // openJARs(): reopens only when the handle was nulled
   if (jarFiles[0] == null) { ... jarFiles[i] = new JarFile(jarRealFiles[i]); ... }
   // getJarFiles() calls openJARs() first — so a *fresh* lookup always gets an OPEN handle

2. The static-serving path captures and reuses a handle without reopening. WebDirContext.lookupFromJars() builds new JarResource(jfEntry.jarFile, jfEntry.jarEntry), and JarResource.streamContent() streams from that captured reference with no liveness
   check:

   InputStream jin = jarFile.getInputStream(jarEntry);   // WebDirContext.java:474

3. ProxyDirContext caches the JarResource for over-threshold entries and never re-resolves it. A resource’s content is cached only
   when contentLength < cacheObjectMaxSize where cacheObjectMaxSize = cacheMaxSize / 20 (default cacheMaxSize 10240 KB ⇒ 512 KB). For
   a larger asset, cacheLoad() still stores the CacheEntry with entry.resource = the live JarResource but with content null. On
   lookup after cacheTTL (default 5000 ms), revalidate() compares only lastModified and contentLength:

   return (lastModified == lastModified2) && (contentLength == contentLength2);

   If unchanged (the JAR on disk hasn’t changed), the same CacheEntry and its stale JarResource are reused — the lookup never goes
   back through WebDirContext/getJarFiles()/openJARs(), so the self-healing reopen in (1) is bypassed.

Net: request #1 caches a JarResource while the JAR is open → the 90 s idle reaper closes (and nulls) the classloader’s handle, but the
cached JarResource still references the now-closed JarFile → request #2 revalidates by timestamp only, reuses the stale JarResource,
and streamContent() calls getInputStream() on a closed JarFile → IllegalStateException: zip file closed.

(An additional, faster trigger for the same end state is a JVM-wide JAR-cache close on (re)deploy via JarFileUtils.closeCachedJarFiles() —
see Payara #5001 — but the idle reaper alone reproduces it with no deploy.)

Why only large resources

Sub-512 KB resources are content-cached as a byte[] at first lookup and served from memory — they never call streamContent() or touch
the live JarFile, so a closed handle is harmless. Only resources above cacheMaxSize/20 stream live from the cached JarResource and hit
the closed handle.

Workarounds (confirm the analysis; not fixes)

• Raise cacheMaxSize (per-app in glassfish-web.xml, e.g. <property name="cacheMaxSize" value="65536"/> → ~3.2 MB per-file cap) so the
  asset is content-cached and never streamed from the JAR. Effective and verified.
• Unpack the webjar into the docroot so the asset is served from the filesystem (no JarFile involved).

Suggested fix

Stop serving from a cached Resource that wraps a borrowed, time-limited JarFile:

• In ProxyDirContext, for entries whose content is not cached (over cacheObjectMaxSize), do not retain/reuse the Resource across
  revalidation — re-resolve via the wrapped DirContext on each access so the lookup goes back through
  WebDirContext.getJarFiles()/openJARs() and obtains a live handle; and/or
• Make WebDirContext$JarResource.streamContent() resilient: if jarFile.getInputStream(jarEntry) fails because the JarFile is closed,
  re-fetch a live handle from the provider (getJarFiles() → openJARs()), re-resolve the entry, and retry.

References

• Payara Web Service screen in web admin should not offer a Test button for JAX-RPC services #5001 — "Payara closing JarFiles that are in use" (same exception via JarFileUtils.closeCachedJarFiles; closed as abandoned)
• Key classes: org.apache.naming.resources.ProxyDirContext (cacheLoad/revalidate, cacheObjectMaxSize = cacheMaxSize/20),
  org.apache.naming.resources.WebDirContext$JarResource (streamContent), org.glassfish.web.loader.WebappClassLoader
  (closeJARs/openJARs, 90 s idle, JarFileResourcesProvider)

[mafaul]

[mafaul]

glassfish-webjar-bug-repro.zip