Extracted file in the same dir as the archive?

[NoLooseEnds]

Thank you for this project, I'm currently looking into to this as a replacement for rar2fs, but "dropping" it in my current workflow unfortunately does not work out of the box because of the different "placement" of extracted files. I use it mainly for symlinks, so this breaks all preexisting symlinks.

Current behaviours:
rar2fs:
/directory/archive.rar -> /directory/extracted.ext

ratarmount:
/directory/archive.rar -> /directory/archive.rar/extracted.ext (Current behavior)

So, ratarmount creates a folder for the archive (named the same as the archive), instead of putting the files toplevel (same dir as the archive it self) like rar2fs.

Also, with multipart rar-files, all the other archive files is still present. This is a listing from a mounted directory containg a multipart archive:

-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r00
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r01
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r02
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r03
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r04
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r05
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r06
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r07
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r08
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r09
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r10
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r11
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r12
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r13
-rwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.r14
-rwxrwxrwx+ 1 950    950   35M Sep  8  2023 archive-testing.r15
drwxrwxrwx+ 1 950    950   96M Sep  8  2023 archive-testing.rar

While archive-testing.rar is a directory that contains the archive file(s).

Ideally, all the rar files should be "gone", and all that is left in the directory is the contents of the archive.

Thanks!

[mxmlnkn]

With what exact options are you calling rar2fs and ratarmount? I don't understand how /directory/archive.rar -> /directory/extracted.ext should work. What if there are multiple RAR files in that directory? Will all their contents be in the same directory? What if there are overlapping file names?

[NoLooseEnds]

Thanks for the fast reply!

This presumes that there is only one archive per directory. If there is multiple, and they have the same filename, it would cause conflict (as you mentioned). That could potentially be resolved by just appending a 1-2-3 or the archive name to the filename? Or just fail, not show the file.

The mount options with rar2fs is just rar2fs -o allow_other – nothing special.
With ratarmount I have ratarmount -f --recursive --lazy -o allow_other --index-folders /var/cache/ratarmount

My dream would be a flag that just say --unwrap, --flatten or something similiar that causes the archives to be shown in the same directory instead of the archive file (also ideally, hide the multi archive files).

EDIT:
The reason I found this awesome project was that rar2fs is not that active any more, and more importantly did something strange that gave different md5 sums of the files. I have a lot of symlinks targeted the rar2fs filesystem (makes it possible to keep multiple filenames and point to an "extracted" archive with a different filename). And I am experimenting with different sync/backup solutions, but it kept uploading the same files (due to the md5 sum differ from time to time on the exact same file). That led me here. Thanks for reading my wall of text. :)

ie.: /mnt/symlinks/completely-different-filename.ext --> /mnt/extracted/some/path/archive.ext

And now with ratarmount the symlinks break, because it is /mnt/symlinks/completely-different-filename.ext --> /mnt/extracted/some/path/archive.rar/archive.ext

Hope that makes sense.

[mxmlnkn]

The mount options with rar2fs is just rar2fs -o allow_other – nothing special. With ratarmount I have ratarmount -f --recursive --lazy -o allow_other --index-folders /var/cache/ratarmount

I have almost forgotten that mounting a folder over itself works. I am not sure how much I would recommend it. If it works, it works.

My dream would be a flag that just say --unwrap, --flatten or something similiar that causes the archives to be shown in the same directory instead of the archive file (also ideally, hide the multi archive files).

There are a lot of options already. I just am not really sure what you want to achieve and are willing to sacrifice for it.

• For starters, instead of --recursive, you could use the default union mount, i.e., ratarmount archive1.rar archive2.rar directory. Then, directory would contain the contents of archive1.rar and archive2.rar without additional subfolders. This option will not work with --lazy though. Not sure if you really need that. Did rar2fs support some --lazy-like functionality?

• There is also the --transform-recursive-mount-point '^.*/([^/]+).rar$' '/' option, but unfortunately, it does not work with folder recursion, yet. It could be made to work, I think. This would probably be the most realistic option if the above does not work for you.

• There is also the -o modules=subdir,subdir=<folder-prefix> option, but this would only work for a single archive, e.g., with directory/folder-symlink.rar ratarmount --recursive -o modules=subdir,subdir=/folder-symlink.rar directory mounted. Now, you won't see the /folder-symlink.rar subfolder.

EDIT: The reason I found this awesome project was that rar2fs is not that active any more, and more importantly did something strange that gave different md5 sums of the files.

Interesting, but sad to hear that development has stopped. The MD5 changes sound like a critical bug.

[NoLooseEnds]

It's hundreds of directories each containing one archive (mostly multipart rar). Imagine /category/name/name.rar, name.r00, name.r01, etc. that mostly extracts to one file. Then I use an "extract" filesystem on top (to skip having to extract the archives) that I then symlinks to (to get other filenames). So instead of "name.ext", it's "something-totally-different.ext".

I've done this with hundreds of archives, so I have the symlinks linked to the extracted file in the same location that the archive is. The issue is that, when I use ratarmount all symlinks break. And I would avoid to have to remake all symlinks.

[mxmlnkn]

It's hundreds of directories each containing one archive (mostly multipart rar). Imagine /category/name/name.rar, name.r00, name.r01, etc. that mostly extracts to one file. Then I use an "extract" filesystem on top (to skip having to extract the archives) that I then symlinks to (to get other filenames). So instead of "name.ext", it's "something-totally-different.ext".

In that case, this should actually work:

ratarmount --transform-recursive-mount-point '^(.*)/([^/]+).rar$' '\1' --recursive /category mounted

E.g., with this test setup:

directory/
├── name-1
│   └── folder-symlink.rar
└── name-2
    └── folder-symlink.rar

With the usual call ratarmount --recursive /category mounted, it would look like this:

mounted
├── name-1
│   └── folder-symlink.rar
│       └── foo
└── name-2
    └── folder-symlink.rar
        └── foo

With ratarmount --transform-recursive-mount-point '^(.*)/([^/]+).rar$' '\1' --recursive directory mounted, you get:

mounted
├── name-1
│   └── foo
└── name-2
    └── foo

Exactly what you want, as far as I understand.

If this does not solve your problem, please provide a minimal example/test reproducer if possible.

• The issue with the multi-part RAR parts showing up is noted. This is a bug / insufficiency.

[NoLooseEnds]

Thanks! That seems to work! :)

It seems like it lags, that when I first list the dir I see all files, but then I just get the extracted file (also not seeing the r00, r01, etc.). I'm guessing that is because of the --lazy flag I use?

[mxmlnkn]

Yes, the --lazy flag moves the processing of the archives when they are first "accessed", e.g., in a directory listing with ls. Without it, all the archives would be analyzed at ratarmount start, but then it would not be possible to dynamically add new RAR archives into the source folder and mount them. I guess there could be some demand for a non-lazy dynamic source folder watch+mount function. These two independent functionalities are currently combined in --lazy.

• Ideally, the first access should already show the extracted archives as folders, though. It is unexpected that you see the files on the first access. I can reproduce this. ls -la mounted/name-1/ yields:

  ls: cannot access 'mounted/name-1/folder-symlink.rar': No such file or directory
  total 0
  -????????? ? ? ? ?            ? folder-symlink.rar
  

  And the next access shows the correct contents.

[NoLooseEnds]

Yes.

And unfortunately now the new issue occurs (due to my workflow). I need to use the --lazy flag or else it gobbles up all the memory and takes way to long time (I tested), but when I use --lazy the symlinks are still broken (until I manually list that dir). And that breaks the sync/backup process (it errors when the symlink target dont exist)

[mxmlnkn]

With the setup you currently described, I do not understand fully why there should be a difference between --lazy and without regarding memory usage, and even time. It is only a matter of when the archives are mounted. At some point, you should end up with the same memory usage.

I think you should try to replace --recursive with --recursion-depth 1. Else, it might be that the files inside those RAR archives are also mounted and analyzed, and given that you are coming from rar2fs, I doubt that is what you want, and it might explain why --lazy supposedly uses less memory than without --lazy.

[NoLooseEnds]

Unfortunately that will not work, since the folder depth varies. I just saw the memory go up, when it parsed through everything. It's a huge NFS share I'm mounting, but not accessing all files. I guess a work around could be to remove --lazy and be more specific with my mount.

Or, is there a way to limit what archives that are supported? Could I limit it to just .rar files for isntance – and keep .zip or .tar (etc.) as is?

[mxmlnkn]

since the folder depth varies.

The recursion depth does not depend on the folder hierarchy. It only accounts for archives. Any archive, no matter how deeply nested it is in a folder, will count as recursion depth 1. Any archive inside that archive will count as recursion depth 2.

Or, is there a way to limit what archives that are supported? Could I limit it to just .rar files for isntance – and keep .zip or .tar (etc.) as is?

• Not yet. I have been wanting this for a long time. It might be time to tackle this.

[NoLooseEnds]

I see. Thanks for all the input. I'll need to check. And that feature would be great for my usecase. :)

I also might have spoken too soon about rar2fs and the md5 issue, since it seems to keep happening. Could be the backup provider that is wonky too (whitelabel jotta cloud).