[JBMAR-240] Missing impl of validation in deSerialization causes memory leak

[dmlloyd]

mitch bass opened JBMAR-240 and commented

jboss-marshalling/tree/main/api/src/main/java/org/jboss/marshalling/cloner]/SerializingCloner.java

implements inner class StepObjectInputStream which extends MarshallerObjectInputStream

 

But registerValidation is not implemented, its an empty function.

 

It is required that it be implemented, in particular javax.swing.JComponent uses it and builds a static internal table(readObjectCallbacks ) that is only cleared of entries when they are used.

not allowing JComponent to clear its readObjectCallbacks table results in a memory leak. the more often this is called the bigger the leak. what winds up being leaked are instances of SerializingCloner and its inner class StepObjectInputStream.

a close review of the java std implementation of ObjectInputStream will show what steps are required to implement a fix.

i've attached a screen shot of the heap memory analyzer output. notice that there is a static table that is taking up over 300MB of space in the JComponent class, named readObjectCallbacks, it has 22K entries, each one of which has as a key the SerializingCloner's inner class StepObjectInputStream object.

---

Attachments:

• SerializingCloner.java.mod2.txt (44.29 kB)
• SerializingClonerMemoryLeak.PNG (176.86 kB)

[dmlloyd]

mitch bass commented

if the nominal solution is to make the class being serialized for cloning cloneable, at least the register validation callback should throw a runtime exception or at the very least log an error.

[dmlloyd]

mitch bass commented

Added modified SerializationCloner.java code. it incorporates the validation list handling from the openjdk implementation of java/io/ObjectInputStream.java. It has been tested for the specific case mentioned in the description.