Add support for better support of scanning single rom based systems

[pasnox]

Is your feature request related to a problem?

Because of the complexity of the various DATs and Systems (Single VS Multiple roms based games) available we can't have a single call of Igir to maintains our roms collection. One have to create manually some scripts and do some dirty works by themselves specifically for cases.
In this feature request I want to optimize the way one can maintains collections for systems only requiring a single rom per game (think, game gear, master system, ...).

I have created a hierarchical layout for Igir that I think suits well for any collections of any systems, only the Igir call for cases would deviate a bit (Single VS Multiple). This is mostly based on the No-Intro Manufacturer / Machine / Variant approach.

This is my layout (shortened):

DATs
DATs/Sega/Game Gear/Sega - Game Gear.dat
DATs/Sega/Game Gear/Aftermarket/Sega - Game Gear (Aftermarket).dat
DATs/Sega/Master System - Mark III/Sega - Master System - Mark III.dat
DATs/Sega/Master System - Mark III/Aftermarket/Sega - Master System - Mark III (Aftermarket).dat

INPUTs
INPUTs/Sega/Game Gear
INPUTs/Sega/Master System - Mark III

OUTPUTs
OUTPUTs/Sega/Game Gear
OUTPUTs/Sega/Game Gear/Aftermarket
OUTPUTs/Sega/Master System - Mark III
OUTPUTs/Sega/Master System - Mark III/Aftermarket

Notice that DATs inner layout is mapped 1:1 into OUTPUTs and DATs inner layout is also mapped 1:1 into INPUTs but excluding variants.
This idea is that DATs and OUTPUTs are well organized and INPUTs allow to throw easily new content to be scanned at the machine level.

One of the major pain I saw is that the --dir-dat-mirror option is based on how we do pass arguments, relative or absolute. In the case of absolute file path then the absolute dir of the dat file is append to the output dir, creating very unwanted nested dirs, especially on unix file systems where there is no drive letters. While the relative one somehow do what we want. Still that's highly dependant from where we do call Igir from and where is located the dat files.
We are lacking a way to tell Igir, hey, from all the various DATs files i passed as argument, consider they have their root dir at X:
igir --dat-root-dir=/absolute/path/to/DATs --dat Manufacturer/Machine/Machine.dat --dat Manufacturer/Machine2/Machine2.dat
This would allow igir to handle correctly relative file paths to dats agnostic to the Igir binary / current working dir location.
This command pass relative DATs according to the dat-root-dir, not according to the current working directory or igir binary location and then allow to expose more replaceable.

The issue is that this kind of collection can't be scanned in one go, mainly because the set of replaceable is under supported.
We are lacking (at a glance) those ones:

• For the dat file
  • CURRENT_DAT_FILE_PATH (absolute file path to .dat file)
  • CURRENT_DAT_RELATIVE_FILE_PATH (relative file path to .dat file)
  • CURRENT_DAT_FILE_NAME (file name of .dat file)
  • CURRENT_DAT_BASE_NAME (base name of .dat file)
• For the directory containing the dat file
  • CURRENT_DAT_DIR_PATH (absolute dir path of .dat file)
  • CURRENT_DAT_RELATIVE_DIR_PATH (relative dir path of .dat file)
  • CURRENT_DAT_DIR_NAME (dir name of .dat file) (path/to/the/file.dat -> 'the')
  • CURRENT_DAT_RELATIVE_DIR_NAME_X_Y (dir name at section X of length Y of .dat file relative directory) (path/to/the/file.dat -> X=1 'to', Y could be optional, -1 would take up to the end)

With those extended replaceable, it would be much more easy to scan everything at once still optimizing the inputs and output to read/write.
Currently I have to do some scripting and call Igir one system at a time so I can optimize inputs/outputs, but to be defeated by the cache to be read/Write anytime.

This is a part of my script to scan / build single based roms sets:

# Igir for each system
for SINGLE_SYSTEM_DAT in "${SINGLE_SYSTEMS_DATS[@]}"; do
    SINGLE_SYSTEM_SET_NAME="$(fileFullBaseName "${SINGLE_SYSTEM_DAT##*/}")"
    SINGLE_SYSTEM_SET_SUBTREE="$(dirname "${SINGLE_SYSTEM_DAT}")"
    SINGLE_SYSTEM_SET_DAT="${DATS_PWD}/${SINGLE_SYSTEM_DAT}"
    SINGLE_SYSTEM_SET_INPUT="${NEWS_PWD}/$(echo "${SINGLE_SYSTEM_SET_SUBTREE}" | cut -d '/' -f 1-2)"
    SINGLE_SYSTEM_SET_OUTPUT="${SETS_PWD}/${SINGLE_SYSTEM_SET_SUBTREE}"
    SINGLE_SYSTEM_SET_REPORT="${SINGLE_SYSTEM_SET_OUTPUT}/#igir_${SINGLE_SYSTEM_SET_NAME}.csv"
    
    echo "##############################################################"
    echo "# Scanning ${SINGLE_SYSTEM_SET_NAME} system..."
    echo "##############################################################"
    echo "SINGLE_SYSTEM_SET_NAME: ${SINGLE_SYSTEM_SET_NAME}"
    echo "SINGLE_SYSTEM_SET_SUBTREE: ${SINGLE_SYSTEM_SET_SUBTREE}"
    echo "SINGLE_SYSTEM_SET_DAT: ${SINGLE_SYSTEM_SET_DAT}"
    echo "SINGLE_SYSTEM_SET_INPUT: ${SINGLE_SYSTEM_SET_INPUT}"
    echo "SINGLE_SYSTEM_SET_OUTPUT: ${SINGLE_SYSTEM_SET_OUTPUT}"
    echo "SINGLE_SYSTEM_SET_REPORT: ${SINGLE_SYSTEM_SET_REPORT}"

    mkdir "${SINGLE_SYSTEM_SET_INPUT}"
    mkdir "${SINGLE_SYSTEM_SET_OUTPUT}"

    igir \
        move \
        zip \
        test \
        --dat "${SINGLE_SYSTEM_SET_DAT}" \
        --input "${SINGLE_SYSTEM_SET_OUTPUT}" \
        --input "${SINGLE_SYSTEM_SET_INPUT}" \
        --output "${SINGLE_SYSTEM_SET_OUTPUT}" \
        --merge-roms split \
        --overwrite-invalid \
        --fix-extension always || true # Continue
done

For Game Gear (Aftermarket), that would translate to something like:

    igir \
        move \
        zip \
        test \
        --dat "$HOME/Igir/DATs/Sega/Game Gear/Aftermarket/Game Gear (Aftermarket).dat" \
        --input "$HOME/Igir/SETs/Sega/Game Gear/Aftermarket" \
        --input "$HOME/Igir/NEWs/Sega/Game Gear" \
        --output "$HOME/Igir/SETs/Sega/Game Gear/Aftermarket" \
        --merge-roms split \
        --overwrite-invalid \
        --fix-extension always

Describe the solution you'd like

If we have the new dat-root-dir parameter and the new replaceable, we could optimize everything in one go:

    igir \
        move \
        zip \
        test \
        --dat-root-dir "$HOME/Igir/DATs"
        # As many as needed single dat files
        --dat "Sega/Game Gear/Game Gear.dat" \
        --dat "Sega/Game Gear/Aftermarket/Game Gear (Aftermarket).dat" \
        --dat "Sega/Master System - Mark III/Master System - Mark III.dat" \
        --dat "Sega/Master System - Mark III/Aftermarket/Master System - Mark III (Aftermarked).dat" \
        # The machine specific output folder as input, $HOME/Igir/SETs/Sega/Game Gear, $HOME/Igir/SETs/Sega/Master System - Mark III/Aftermarket, ...
        --input "$HOME/Igir/SETs/{CURRENT_DAT_RELATIVE_DIR_PATH}" \
        # The machine specific input, $HOME/Igir/NEWs/Sega/Game Gear, $HOME/Igir/SETs/Sega/Master System - Mark III
        --input "$HOME/Igir/NEWs/{CURRENT_DAT_RELATIVE_DIR_NAME_0_2}" \
        # The machine specific output, $HOME/Igir/SETs/Sega/Game Gear, $HOME/Igir/SETs/Sega/Master System - Mark III/Aftermarket, ...
        --output "$HOME/Igir/SETs/${CURRENT_DAT_RELATIVE_DIR_PATH}" \
        --merge-roms split \
        --overwrite-invalid \
        --fix-extension always

Additional context

With those enhancements we would be allowed to really batch easily multiple scans for the same kind of sources, all optimized so that the input only checks for roms data that matter to the current dat instead of global pool of all roms.

[emmercm]

Igir does internally determine a DAT root directory, as you mentioned, and uses it for processing such as determining where to put dir2dats. It's not currently configurable, and I'm not yet sure it has to be (or that I can sufficiently explain what it means to the user).

--dir-dat-mirror not handling relative vs. absolute --dat <path> the same is an oversight. I would argue it should behave the same either way, and that no one would want their entire absolute path duplicated in --output <path>. I can fix this.

I think #2431 should address some of the other concerns you shared here.

[emmercm]

Can you provide an example of when --dir-dat-mirror handled relative vs. absolute paths differently? I haven't been able to reproduce it yet.

[pasnox]

I think but not sure (not on the machine at this time):

igir move zip --dir-dat-mirror --dat /absolute/path/to/some/dat/file.dat...
would be enough so that the output dir would become:
OUTPUT/absolute/path/to/some/dat/

if it's not reproducible, try again with 2 absolute --dat parameters.

This is on linux, but it probably fail the same for any unix like file system.
Windows would be immune because of the driver later thing i would say.