Skip to content

Wrongly converting columns to nullable may cause momory problem #431

@shuai-xu

Description

@shuai-xu

Describe what's wrong
When I run a query with gouping set, It failed with exception java.lang.UnsupportedOperationException: Not enough spark off-heap execution memory. Acquired: 17592186700480, granted: 2143226266. The stack shows as following:
image
It took a long time for me to find that this is due to wrongly convert columns to nullable in ExpandTransform. This action may cause that data in Block is nullable, but the header in the next transform is not nullable, when seriaze the data and then deseriaze it, it will throw exception.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions