Regarding the data processing issues after performing overlap in OpenMX

[fffffreeeeee]

Hello, I have now used OpenMX to perform the overlap calculation, then used DeepBack to convert the data, and later I intended to use DeepDock to upgrade the data. However, after converting the data, the folder contains the following contents.
overlapdata/0:
config.ini hamiltonians.h5 log openmx.dat openmx_rst openmx.xyz output POSCAR site_positions.dat
element.dat lat.dat openmx.cif openmx.out openmx.scfout orbital_types.dat overlaps.h5 rlat.dat

/0/openmx_rst:
openmx.ftCvna openmx.ftnlp openmx.ftpao openmx.ftPEvna openmx.ftProPAO

/0/output:
overlaps_0.h5 overlaps_17.h5 overlaps_24.h5 overlaps_31.h5 overlaps_39.h5 overlaps_46.h5 overlaps_53.h5 overlaps_6.h5
overlaps_10.h5 overlaps_18.h5 overlaps_25.h5 overlaps_32.h5 overlaps_3.h5 overlaps_47.h5 overlaps_54.h5 overlaps_7.h5
overlaps_11.h5 overlaps_19.h5 overlaps_26.h5 overlaps_33.h5 overlaps_40.h5 overlaps_48.h5 overlaps_55.h5 overlaps_8.h5
overlaps_12.h5 overlaps_1.h5 overlaps_27.h5 overlaps_34.h5 overlaps_41.h5 overlaps_49.h5 overlaps_56.h5 overlaps_9.h5
overlaps_13.h5 overlaps_20.h5 overlaps_28.h5 overlaps_35.h5 overlaps_42.h5 overlaps_4.h5 overlaps_57.h5
overlaps_14.h5 overlaps_21.h5 overlaps_29.h5 overlaps_36.h5 overlaps_43.h5 overlaps_50.h5 overlaps_58.h5
overlaps_15.h5 overlaps_22.h5 overlaps_2.h5 overlaps_37.h5 overlaps_44.h5 overlaps_51.h5 overlaps_59.h5
overlaps_16.h5 overlaps_23.h5 overlaps_30.h5 overlaps_38.h5 overlaps_45.h5 overlaps_52.h5 overlaps_5.h5
But after I ran the command "dock convert deeph upgrade -t -1 --force ./overlapdata/0 ./new_overlapdata", an error occurred.
Data: 0it [00:00, ?it/s]Skip . because of missing necessary files.
Data: 0it [00:00, ?it/s]
[done] Translation completed successfully!
I would like to know what the necessary file is for this conversion to fail. I want to know what the necessary file is and how to solve this problem.I am looking forward to your reply.

[kYangLi]

Subject: Re: Issue with dock convert deeph upgrade command

Hello,

Thank you for providing such detailed information about your data structure and the error message. I can see what's happening here.

The issue is related to the folder structure of your data. Looking at your directory layout:

overlapdata/0:
config.ini hamiltonians.h5 log openmx.dat openmx_rst openmx.xyz output POSCAR site_positions.dat
element.dat lat.dat openmx.cif openmx.out openmx.scfout orbital_types.dat overlaps.h5 rlat.dat

/0/output:
overlaps_0.h5 overlaps_17.h5 ... (many individual overlap files)

The dock convert deeph upgrade command expects each data sample (structure) to be in a separate folder, and each folder should contain the necessary files directly, not in subdirectories.

The Problem:
Your data is currently organized with all files in one folder (overlapdata/0), including an output/ subdirectory containing multiple overlaps_*.h5 files. The conversion script is looking for specific files in the main folder and might be confused by the output/ subdirectory or expecting a different structure.

The Solution:
You need to reorganize your data according to the expected folder structure. Each structure should have its own folder, and within that folder, you should have the necessary files directly (not in subfolders).

Expected structure should look like:

overlapdata/
├── structure_1/
│   ├── hamiltonians.h5
│   ├── overlaps.h5
│   ├── config.ini
│   ├── element.dat
│   ├── lat.dat
│   ├── rlat.dat
│   ├── site_positions.dat
│   └── orbital_types.dat
├── structure_2/
│   ├── hamiltonians.h5
│   ├── overlaps.h5
│   └── ...
└── ...

What you should do:

1. Move the necessary files from overlapdata/0/output/ to overlapdata/0/ if they are required (though typically overlaps.h5 in the main folder should suffice).

2. If you have multiple structures, create separate folders for each one.

3. Ensure each folder contains at minimum:

   • hamiltonians.h5 or overlaps.h5
   • config.ini
   • element.dat
   • lat.dat, rlat.dat
   • site_positions.dat
   • orbital_types.dat

Reference:
For more details on the expected folder structure, please refer to the official documentation:
https://docs.deeph-pack.com/deeph-dock/en/latest/key_concepts.html#folder-structure

Additional Note:
If your overlaps.h5 file in the main directory already contains all the overlap information, you might not need the individual overlaps_*.h5 files in the output/ subdirectory at all. You could try removing or moving the output/ folder and then running the conversion command again.

Please reorganize your data according to this structure and try the conversion command again. If you continue to encounter issues, please let me know the exact structure of your folders after reorganization.

Best regards,
The DeepH Team

[fffffreeeeee]

Subject: Re: Issue with dock convert deeph upgrade command

Hello,

Thank you for providing such detailed information about your data structure and the error message. I can see what's happening here.

The issue is related to the folder structure of your data. Looking at your directory layout:

overlapdata/0:
config.ini hamiltonians.h5 log openmx.dat openmx_rst openmx.xyz output POSCAR site_positions.dat
element.dat lat.dat openmx.cif openmx.out openmx.scfout orbital_types.dat overlaps.h5 rlat.dat

/0/output:
overlaps_0.h5 overlaps_17.h5 ... (many individual overlap files)

The dock convert deeph upgrade command expects each data sample (structure) to be in a separate folder, and each folder should contain the necessary files directly, not in subdirectories.

The Problem: Your data is currently organized with all files in one folder (overlapdata/0), including an output/ subdirectory containing multiple overlaps_*.h5 files. The conversion script is looking for specific files in the main folder and might be confused by the output/ subdirectory or expecting a different structure.

The Solution: You need to reorganize your data according to the expected folder structure. Each structure should have its own folder, and within that folder, you should have the necessary files directly (not in subfolders).

Expected structure should look like:

overlapdata/
├── structure_1/
│   ├── hamiltonians.h5
│   ├── overlaps.h5
│   ├── config.ini
│   ├── element.dat
│   ├── lat.dat
│   ├── rlat.dat
│   ├── site_positions.dat
│   └── orbital_types.dat
├── structure_2/
│   ├── hamiltonians.h5
│   ├── overlaps.h5
│   └── ...
└── ...

What you should do:

1. Move the necessary files from overlapdata/0/output/ to overlapdata/0/ if they are required (though typically overlaps.h5 in the main folder should suffice).

2. If you have multiple structures, create separate folders for each one.

3. Ensure each folder contains at minimum:

   • hamiltonians.h5 or overlaps.h5
   • config.ini
   • element.dat
   • lat.dat, rlat.dat
   • site_positions.dat
   • orbital_types.dat

Reference: For more details on the expected folder structure, please refer to the official documentation: https://docs.deeph-pack.com/deeph-dock/en/latest/key_concepts.html#folder-structure

Additional Note: If your overlaps.h5 file in the main directory already contains all the overlap information, you might not need the individual overlaps_*.h5 files in the output/ subdirectory at all. You could try removing or moving the output/ folder and then running the conversion command again.

Please reorganize your data according to this structure and try the conversion command again. If you continue to encounter issues, please let me know the exact structure of your folders after reorganization.

Best regards, The DeepH Team

Hello, I have already formatted the files into the corresponding format. I have placed them in /pro-overlapdata/0. It contains the files
config.ini, element.dat, lat.dat, orbital_types.dat, overlaps.h5, rlat.dat, and site_positions.dat.
I ran the command "dock convert deeph upgrade ./pro-overlapdata/0 ./0 -p 2". The prompt for the command was:
(deeph) (base) [user200@gpu1 tutorial_DeepH-E3]$ dock convert deeph upgrade ./pro-overlapdata/0 ./0 -p 2 Data: 0it [00:00, ?it/s]
[done] Translation completed successfully!
However, there is no output in the 0 folder. Could you please tell me what the problem is?