Skip to content

Improve struct dictionary codecs in Go #153

@tigrannajaryan

Description

@tigrannajaryan

We do a lot of unnecessary copying and cloning of dictionary-encoded structs currently, especially during encoding.

Consider the following:

  • Allow marking structs frozen. Once frozen they can be safely referenced by pointer from multiple places (including dictionaries) without cloning.
  • Use freezing after record is read.
  • Allowing referencing frozen structs from read operations directly in write operations. Should improve copying (read/write).
  • Make CopyFrom smarter, avoid cloning frozen structs, just copy the pointer.
  • Encourage Freeze() when creating data (not from Read()) of Writer. Should improve encoding.
  • Cache refNum in the object to avoid slow Tree lookups.
  • Replace Tree by Hashmap. Cache the hashcode, invalidate on modifications.
  • Allowing setting dictionary structs in their containing structs by pointer instead of requiring full CopyFrom or field-by-field setting.

Most of these can be applied to varying degrees both to Go and Java. This summary issue is for Go. Duplicate it for Java.

See some initial experiment on this branch https://github.com/splunk/stef/compare/main...tigran/profileserialize?expand=1

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions