Technical Limitation --> StorageEntityCache.java

[lipoeAcc]

While testing EclipseStore, I encountered a storage exception that I am struggling to fully understand. The specific exception is as follows:
throw new StorageException(
"Storage position for entity " + entity.objectId()
+ " exceeds the technical int value limit of " + Integer.MAX_VALUE + "."
+ " This happens when a single store grows too big."
+ " This limitation will be removed in a future version."
);

It seems to reference a hardcoded maximum value for the storage offset, which leads to an error if the store grows beyond Integer.MAX_VALUE:.

I also came across a comment (see the picture) regarding "storage management 2.0" as a potential improvement or alternative, but the comment appears to be several years old.

Could someone clarify whether this limitation is still necessary in the current version? If it is still necessary, what is the reason for this limitation? Additionally, is the mentioned approach of working with multiple files still viable, or have there been other improvements in this area?

Thank you in advance for your time and support.

[fh-ms]

Like the exception states, the data amount for a single store was too big.

This is a technical limitation of the current storage engine. The next generation of the engine will eliminate this problem, but it's still in an experimental stage.

For now, the solution would be to split this single store operation into multiple ones.

If you can share some code, we should find a suitable solution.

[lipoeAcc]

Thank you for clarifying the limitation and providing a workaround. It's good to know that the issue is being addressed in the next generation of the storage engine.

Do you happen to know if there’s a timeline or estimated release date for this new engine? Considering the comment I came across is from more than four years ago, I’m curious about the current status of the development.

I appreciate your guidance on splitting the store operations. My testing is focused on evaluating EclipseStore’s long-term feasibility as a database solution, so there's no need for a concrete implementation at this stage. However, your offer to help is greatly appreciated!

[hrstoyanov]

@fh-ms @hg-ms
Will "the next generation" mentioned above be part of the open source Eclipse Store project, or like GigaMap? And if it open source, is there a branch or repo to look at?

[fh-ms]

Currently, it is a closed-source project. It's not yet decided how it will be published.

[hrstoyanov]

@fh-ms, publishing some roadmap for what is coming up in ES will help build trust with the community. Not disclosing future plans makes me very reluctant to invest further time, effort and contributions to ES.

[lipoeAcc]

I also noticed behavior that might be relevant to this topic and could potentially indicate a bug.

While experimenting with the dataFileMaximumSize configuration, I observed the following:

EmbeddedStorageManager storage = EmbeddedStorageConfigurationBuilder.New() .setDataFileMaximumSize(ByteSize.New(2146535072 , ByteUnit.B)) .setChannelCount(4) .setChannelDirectoryPrefix("channel_") .setDataFilePrefix("channel_") .setStorageDirectory(this.nioFileSystem.ensureDirectoryPath(path).toPathString()).createEmbeddedStorageFoundation().createEmbeddedStorageManager();

When setting a high dataFileMaximumSize, the storage engine appears to reach its technical limit prematurely. Interestingly, this happens inconsistently, with failures occurring at different data sizes despite identical settings. Additionally, this issue only manifests when storing many objects individually rather than using .storeAll().

I’ve attached a graphic summarizing my test results. These focus particularly on the critical range of dataFileMaximumSize values where the issue occurs. Notably, all smaller values for the dataFileMaximumSize were successful.

Although I don’t require a solution to this problem, I wanted to bring it to your attention, as it could be relevant to addressing the technical limitation or the new storage engine development.