java.nio.file.FileSystemException:Too many open files plus MASSIVE storage space used?? #373
-
|
So doing some RAND on EclipseStore. have a relatively simple object (from dtd): where first are just enums. So test: Create 150,000 objects in a tight loop saving after each insertion (simulating events being created and added to the store) So 2 observations: a. 143,000 object takes up 86 GB of storage!! Is this by design?? We are looking at putting millions of objects in the store would that take Terabytes of storage ??? The Root is a simple container object that just has a list of these event objects. Also almost every run after about 143,000 items are inserted we get too many open files exceptions. Is there a flush that needs to be called?? What am I missing ?? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 11 replies
-
|
Hi, |
Beta Was this translation helpful? Give feedback.
-
|
The difference is that if you take a standard collection in Java, add an element to it, and then save this collection, the entire collection is stored as a new version and added to the latest file. Subsequently, there are two processes in Eclipse Store that handle this. The first is housekeeping, which reorganizes these files into regularly sized segments (essentially performing cleanup). The second is the Eclipse Store garbage collector, which removes previously stored objects that are no longer relevant—specifically, the previous versions of this collection. Eclipse Store is designed so that it does not run these processes while data is being saved. Instead, it waits until writing operations are inactive before initiating cleanup. However, if there is no such idle time, it is advisable to use Adaptive Housekeeping and manually allocate enough time for it. In general, 150,000 elements should not be an issue, but it depends on what exactly you are storing. |
Beta Was this translation helpful? Give feedback.
-
|
Ah ok so I am doing this in a tight loop so therefore: Eclipse Store is designed so that it does not run these processes while data is being saved. Instead, it waits until writing operations are inactive before initiating cleanup. There is no time for housekeeping to get invoked correct? However, if there is no such idle time, it is advisable to use Adaptive Housekeeping and manually allocate enough time for it. Is there an example of this? I am going to try and issue: issueFullCacheCheck() every 25,000 list additions and see what happens .. |
Beta Was this translation helpful? Give feedback.
-
|
Update doing the following every 25,000 writes: if (amount % 25000 == 0) { No problems doing 150,000 now. Also noticed performance increase after doing this. That being said doing all those seems a little harsh Please provide an example of Adaptive housekeeping if you can! Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Ok, it's not entirely clear in the documentation—we need to improve that. Nevertheless, if you want to use adaptive housekeeping, you first create a foundation, where you replace the housekeeping controller with an AdaptiveHouseKeepingController. Then, you create the storage from this foundation. I'm sending you a code snippet from a performance test. Of course, the values need to be adjusted according to your specific needs. It's usually best to experiment a little—just set the aggressiveness according to the required scenario. private static EmbeddedStorageFoundation<?> prepareStorageFoundation() {
final EmbeddedStorageFoundation<?> foundation = EmbeddedStorage.Foundation().addEventLogger(new StorageEventLoggerSLF4J());
final StorageHousekeepingController housekeepingController = StorageHousekeepingController.AdaptiveBuilder(foundation)
.increaseAmountNs(100)
.increaseThresholdMs(1_000)
.maximumTimeBudgetNs(50_000)
.buildFor(foundation);
foundation.setConfiguration(
Storage.ConfigurationBuilder()
.setHousekeepingController(housekeepingController)
.createConfiguration()
);
return foundation;
}
private static EmbeddedStorageManager startStorage(final Root root) {
return prepareStorageFoundation().start(root);
}You can then enable logging, and you will see the behavior there. <logger name="org.eclipse.store.storage.types.StorageHousekeepingController$Adaptive" level="DEBUG" /> |
Beta Was this translation helpful? Give feedback.
-
This is not necessarily a bad thing. We use this approach in our projects as well. |
Beta Was this translation helpful? Give feedback.
-
|
Oh ok thank you good to know! I will take a look at your adaptive one as well... Cheers! |
Beta Was this translation helpful? Give feedback.
Ok, it's not entirely clear in the documentation—we need to improve that.
Nevertheless, if you want to use adaptive housekeeping, you first create a foundation, where you replace the housekeeping controller with an AdaptiveHouseKeepingController. Then, you create the storage from this foundation.
I'm sending you a code snippet from a performance test. Of course, the values need to be adjusted according to your specific needs. It's usually best to experiment a little—just set the aggressiveness according to the required scenario.