Skip to content

Improve JsonBlob to prevent OOM errors #455

@cnrudd

Description

@cnrudd

When large amounts of data are stored in the value field of the JsonBlob table, this can cause the Admin/UserData/JsonBlob panel to cause the Grails server to fail with an OOM error when it loads. This can happen pretty quickly with the default memory settings:
tomcat:
requests:
cpu: 1000m
memory: 1Gi
limits:
cpu: 2000m
memory: 2Gi

Raising memory allocations would just be a bandaid.

It might be time to:

Rewrite RestGrid API so that we can specify which fields to NOT load (value in this case) when getting the full list of records, and then provide an API for searching specific fields from the server (value in this case), and also make sure all fields are loaded when an individual record is loaded for editing.

I think this would give the Hoist Admin JsonBlob UI a lot of runway into the future, until we needed to rework the UI again to load rows just in time or with pagination.

Alternatively, we could implement a streaming response controller, in which the blobs were serialized not all at once but instead as they are streamed up to the client. This would probably eliminate server OOM errors, but that would not reduce the strain on the UI which could get sluggish with a lot of data loaded into the grid all at once, and the pressure to rework the UI to load rows just in time or with pagination would come sooner.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions