OrientDB 3.2.42 - Issues with importing data

[tokmgr]

After the upgrade of OrientDB from version 3.0.29 to 3.2.42, issues related to the importing of data from 3.0.29 (created with the
EXPORT command) into an empty database in 3.2.42 were encountered:

• Using the command import database PATH/NAME.gz did import the data from the 3.0.29 export but did not maintain the relationships between entries in a table - the database contains a table where an entry references its parent and the list of children, those relationships were not maintained. To maintain the relationships, the flag -preserveRIDs had to be added to the call. The import statement in 3.0.29, never showed such a behavior.

• Adding the flag -preserveRIDs to the import statement did result in out-of-memory errors at times - no such error was encountered when the flag was not used - but then, the data was not usable as per the previously mentioned issue. Here is the error stacktrace from the server log:

  2025-08-14 12:36:59:460 INFO  {db=db42} Starting database import [OConnectionBinaryExecutor]java.lang.OutOfMemoryError: Java heap space
   Dumping heap to java_pid19136.hprof ...
   Heap dump file created [2423604008 bytes in 45.163 secs]

   Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "OrientDB Timer[embedded:C:\orientdb-community-3.2.42\databases]"
   Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "OrientDB WAL Flush Task"
   Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-0"
   Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "OrientDB Write Cache Flush Task"

  2025-08-14 12:40:54:506 SEVER Uncaught exception in thread OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:54533) [OUncaughtExceptionHandler]
    java.lang.OutOfMemoryError: Java heap space

• Using the class ODatabaseImport in a Java program to import data exported from a 3.0.29 server did not work in any case tried, with or without the call to setPreserveRids(true). In all cases, there were OrientDB internal errors:

  Without the flag:
  
  Importing records...
    > Error on database import happened just before line 0, column 2201774
com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot DELETE the record #0:0 because the version is not the latest. Probably you are deleting an old record or it has been modified by another user (db=v1 your=v0)      DB name="db3"      Error Code="3"  DB name="db3"  Error Code="3"
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleException(OChannelBinaryAsynchClient.java:354)
        at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:302)
        at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:324)
        at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:208)
        at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:167)
        at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2004)
        at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:436)
        at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:501)
        at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:416)
        at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationNoRetry(OStorageRemote.java:451)
        at com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:1367)
        at com.orientechnologies.orient.client.remote.db.document.ODatabaseDocumentRemote.internalCommit(ODatabaseDocumentRemote.java:1171)
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:651)
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:116)
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:94)
        at com.orientechnologies.orient.client.remote.db.document.ODatabaseDocumentRemote.executeDeleteRecord(ODatabaseDocumentRemote.java:564)
        at com.orientechnologies.orient.core.tx.OTransactionNoTx.deleteRecord(OTransactionNoTx.java:246)
        at com.orientechnologies.orient.client.remote.db.document.ODatabaseDocumentRemote.delete(ODatabaseDocumentRemote.java:888)
        at com.orientechnologies.orient.client.remote.db.document.ODatabaseDocumentRemote.delete(ODatabaseDocumentRemote.java:115)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.delete(ODatabaseDocumentAbstract.java:800)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.delete(ODatabaseDocumentAbstract.java:80)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1461)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:232)
        at net.superna.OrientDBManager.OrientDBManager.execute(OrientDBManager.java:629)
        at net.superna.OrientDBManager.OrientDBManager.execute(OrientDBManager.java:457)
		        Suppressed: com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot DELETE the record #0:0 because the version is not the latest. Probably you are deleting an old record or it has been modified by another user (db=v1 your=v0)  DB name="db3"      Error Code="3"
                at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doDeleteRecord(OAbstractPaginatedStorage.java:5135)
                at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:5724)
                at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:2484)
                at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:2309)
                at com.orientechnologies.orient.core.db.document.ODatabaseDocumentEmbedded.internalCommit(ODatabaseDocumentEmbedded.java:1958)
                at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:651)
                at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:116)
                at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:1599)
                at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:1569)
                at com.orientechnologies.orient.server.OConnectionBinaryExecutor.executeCommit38(OConnectionBinaryExecutor.java:1555)
                at com.orientechnologies.orient.client.remote.message.OCommit38Request.execute(OCommit38Request.java:141)
                at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:354)
                at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:238)
                at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:68)


  With the flag set:

Importing records...
Aug 11, 2025 4:10:58 PM com.orientechnologies.common.log.OLogManager log
SEVERE: Error importing record . Source line 0, column 58286 content {"@type":"b","@rid":"#3:0","@version":1,"value":"H4sIAAAAAAAA/+1T0WrjMBD8FaFnOzhtcm0NfhCykhM4lpHWvfrqEEyjtOZcKVhu+xDy73Xia6El+YK7F4FmZmdnF3YXc5UlpEjJgkWKkhVbZFCseBpzSkBIDykO7DybCQlRWje9rkjpXIo8Oy2kJKF5QoDFCgjk6rSKJMBkSoDfMp5Fuflj7JvxEEuBQwFFdiYF8Pg0cT44V4rRXPa2JCFyEW2qxum+QACjIh6K0jxJvqziJ0tmn5bf2YOjvOWUkdmMUeDp/MO0F7C5kEUk1a8gGF/2Ton4uzZ2B5LEBEi0w7XZWBzeY9rqqtPImWrrnmyH6sfG+SSe+lXT1M6aUaddd+H/tkZf+LOqbuyrbtVT1Wp/fL0K/PHVdDK9ubr5MZlcjkdIdVX34kJ0TDNC1K51iKZB4CHdtrYNUWl2pSnx8edKjA7IfY8M8EOvH0BMGF2xO8oy4CItsXegn7Vz1eOH4jPzwzDDpk+n1yESW91WXW0NMj3pXrZb23Z6XeLS7EuzPL7Y+z/6Pzj6crncD4cXM0UlP7Y5c2F8kREKw13t3wHpNQgFvAQAAA=="}
java.lang.UnsupportedOperationException
        at com.orientechnologies.orient.client.remote.db.document.ODatabaseDocumentRemote.commitPreallocate(ODatabaseDocumentRemote.java:1292)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1329)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1423)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:232)

    > Error importing record: 'null' content:'{"@type":"b","@rid":"#3:0","@version":1,"value":"H4sIAAAAAAAA/+1T0WrjMBD8FaFnOzhtcm0NfhCykhM4lpHWvfrqEEyjtOZcKVhu+xDy73Xia6El+YK7F4FmZmdnF3YXc5UlpEjJgkWKkhVbZFCseBpzSkBIDykO7DybCQlRWje9rkjpXIo8Oy2kJKF5QoDFCgjk6rSKJMBkSoDfMp5Fuflj7JvxEEuBQwFFdiYF8Pg0cT44V4rRXPa2JCFyEW2qxum+QACjIh6K0jxJvqziJ0tmn5bf2YOjvOWUkdmMUeDp/MO0F7C5kEUk1a8gGF/2Ton4uzZ2B5LEBEi0w7XZWBzeY9rqqtPImWrrnmyH6sfG+SSe+lXT1M6aUaddd+H/tkZf+LOqbuyrbtVT1Wp/fL0K/PHVdDK9ubr5MZlcjkdIdVX34kJ0TDNC1K51iKZB4CHdtrYNUWl2pSnx8edKjA7IfY8M8EOvH0BMGF2xO8oy4CItsXegn7Vz1eOH4jPzwzDDpk+n1yESW91WXW0NMj3pXrZb23Z6XeLS7EuzPL7Y+z/6Pzj6crncD4cXM0UlP7Y5c2F8kREKw13t3wHpNQgFvAQAAA=="}'

    > Error on database import happened just before line 0, column 58286
java.lang.UnsupportedOperationException
        at com.orientechnologies.orient.client.remote.db.document.ODatabaseDocumentRemote.commitPreallocate(ODatabaseDocumentRemote.java:1292)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1329)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1423)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:232)

Note: In databases using the 3.0.29 version, the binary table is used to store compressed data and link it to entries in other tables - #3:x is the binary table in those cases.

Any feedback would be welcome.

[tglman]

Hi @tokmgr,
If the database was created with version 3.0.x you should be able to open it with 3.2.x without need of the export/import, just copy the database folder in the new distribution or do a normal backup restore.

In any case the links when imported without -PreserveRIDs should be converted automatically, if that do not happen it may be that the record ids are stored in a data type that is not link (like String) so you should double check that.

I need to investicate for the out of memory errors with -PreserveRIDs, I guess you already though to increase the heap size, if you have memory dumps somewhere I would be happy to get some.

If trying to use the import trough java APIs from remote use ODatabaseImportRemote instead ODatabaseImport you cannot run the import logic in remote, it need to send the whole import to the server, if you want to debug the import yourself run the import using an embedded database.

by binary table do you mean ORecordBytes AKA OBlob or a document with inside a byte[] field ? yes this may be the reason why the import do not convert the ids

[tokmgr]

Hi @tglman

Thanks for the quick response.

Unfortunately, copying the database folder is not an option, the database to load is available as an export gz file only, the import has to work.

The table in question for the parent-children relationship issue is using the schemaless setup. This means the values are set as shown next:

  doc.field("children", children);
  doc.field("parent", parent);

where children is of type List<ODocument> and parent is of type ODocument. In the 3.0.29 version, that worked just fine and the relationships were maintained when importing without the flag.

I will try to work with the recommended class and let you know what happened.

And yes, the binary "table" is ORecordBytes.

The OOM issue, yes, we considered the heap dump but then, what value is sufficient? If set to the value x it may work for most imports but eventually there is one for which that setting does not work either. Checking on memory dumps.

[tokmgr]

@tglman

Tried the import using the class ODatabaseImportRemote and the result is worse than before as with or without the below instruction relationships are not maintained:

imp.setPreserveRids(true); // imp being of type ODatabaseImportRemote

The Java version used is 17. Work is now concentrating on the heap size change and use of the IMPORT statement although the preferred approach would be to use the Java APIs. Again, as pointed out before, all was working fine in 3.0.29.

[tglman]

@tokmgr

It may help considering the -deleteRIDMapping=false it does not change the outcome, but will keep a class with the name ___exportImportRIDMap that has two fields key with the original record id and value that will point to the new record, this may help you with data re-mapping.

Another option is if you can produce a minimal export with the problem and point me to the missing links I can try to reproduce and fix the problem from my side

[tokmgr]

@tglman

Will try it with the proposed flag and also get an export together that you can use for your testing. Thanks for the quick replies to this issue.

[tokmgr]

@tglman

I have an export file with which the import command can be tested. Please let me know where to send it to.

Another issue with the import that started showing is this error:

com.orientechnologies.orient.core.exception.ODatabaseException: Impossible to remove the last cluster of class 'ORole' drop the class instead

This one is infrequent.

[tglman]

Hi @tokmgr,

you can send it to tglman at orientdb.dev

[tokmgr]

@tglman

Database send. Thanks for the help on this.

[sblommers]

Hi @tokmgr I also have issues with importing (but also migrating) a database export into 3.2.43 and (now) 3.2.44 but this one is from version 2.2.36 so a much needed upgrade. Also distributed import didn't work.

So my next step is to try plocal connection from console and that seems to work. So i create the database and shutdown orientdb server. After that I just run everything on console->plocal. This imported and migrated everything and it also worked for my indexes. 80GB database

Hope it helps to get you further. And I also hope @tglman can help you with your use case.
Imports/migrations have always been hard on me :-)

Edit: to tame the memory usage of the import process and allowing it to also migrate the indexes I had to change some memory settings in console.sh, I took a shot with the below settings and got succes, not sure what actually made it pass (it was OOM killed a few times before with the defaults)

ORIENTDB_OPTS_MEMORY="-Xms6G -Xmx6G -Dstorage.diskCache.bufferSize=16384 -XX:MaxDirectMemorySize=16g -XX:+PerfDisableSharedMem -Dtx.useLog=false -Dstorage.wal.syncOnPageFlush=false -Dstorage.useWAL=false"

[tglman]

Hi @tokmgr,

I tried to import the database, I didn't see any significant error, usually when the id conversion is missing the id point to nothing, one thing to be aware is that the import may change a bit the default order of the data (ordered by id), but from a quick look does not seems wrong, the record are linking to other record of the correct class type, I did execute the query you provided with the data, and yes I see some different ordering, but to me it feel just like a data result ordering problem and the data is actually correct.

[tokmgr]

Hi @tglman

OrientrDB is in use here for years and we never had an issue with importing a database using the Java API or the import command in that the expected parent-child relationship changed until the recent upgrade to 3.2.42, the correct relationship is crucial. If the query I provided returned the same value on all qualified lines, then this would be correct, otherwise incorrect. If it is not possible to count on this, we have to find a different approach.