When running PCFG in a distributed fashion, the state shared between nodes can become quite large. It contains the full priority queue used to generate items in probability order.
Current serialization does no real compression, except for using variable integers where possible, and using references to base structures instead of serializing themselves. This helps, but when we get far into the keyspace, the size can still run into multiple GiB's. Downsides of this, especially when running with multiple nodes:
- network pressure
- nodes need to wait longer
- larger storage requirements
We can take a look at the possibility of compressing the state before sharing it. Make sure the compression does not take too long, nor uses too much (heap) memory in itself.
Other option is to bound/trim the queue while running, but let's keep it as a backup option.
When running PCFG in a distributed fashion, the state shared between nodes can become quite large. It contains the full priority queue used to generate items in probability order.
Current serialization does no real compression, except for using variable integers where possible, and using references to base structures instead of serializing themselves. This helps, but when we get far into the keyspace, the size can still run into multiple GiB's. Downsides of this, especially when running with multiple nodes:
We can take a look at the possibility of compressing the state before sharing it. Make sure the compression does not take too long, nor uses too much (heap) memory in itself.
Other option is to bound/trim the queue while running, but let's keep it as a backup option.