[Bug]: Stale audioFiles inode makes downloads 404 permanently and no scan repairs it #5568

What happened?

Downloading a book from a mobile app fails with "file not found" while streaming the same book works. The file is on disk, unchanged, and does play when accessed manually.

Audiobookshelf stores each audio file's inode twice, once in libraryItems.libraryFiles[].ino and then again in books.audioFiles[].ino. Remote clients download with GET /api/items/:id/file/:ino/download pulling the ino from media.audioFiles, and the handler compares that against libraryFiles. When the two disagree, the lookup misses and the download 404s.

On my server, annoyingly, they disagree for every audio file that came in during the initial library scan:

broken item, same relPath in both places
  books.audioFiles[0].ino          14395
  libraryItems.libraryFiles[].ino  281474976727661
  stat on disk                     281474976727661

working item, added months later
  all three                        281474976763149

GET /api/items/<broken>/file/14395/download            -> 404
GET /api/items/<working>/file/281474976763149/download -> 206

I cannot tell you how the two got out of step, I have genuinely no idea how that could have happened. I thought it was a drivepool error, but it doesn't seem like it. compareUpdateLibraryFile does update libraryFile.ino when it changes and marks the file modified, which should carry the new value through to the book, so a scan that runs to completion ought to keep them together, but I've run multiple full scans of the library since then, and it was never caught. The two rows are saved separately, so if a scan died between the two saves it could cause this state, but I have no log proving that is what happened and I can't reproduce it consistently on an unmodified server.

I can show that once they have diverged, for whatever reason, no amount of full scanning will bring them back. You have to do the Plex dance to get it to work at scale.

POST /api/items/<broken>/scan  ->  {"result":"UPTODATE"}

The audio file block in BookScanner.rescanExistingBookLibraryItem sits behind one condition:

let hasMediaChanges = libraryItemData.hasAudioFileChanges || libraryItemData.audioLibraryFiles.length !== media.audioFiles.length

hasAudioFileChanges only updates if something changes between the last two scans. libraryFiles already holds the correct value, so every additional scan sees no changes. A library scan with force reaches rescanLibraryItemMedia but that condition still fails to update the ino. An ordinary scan of an unchanged item does not use BookScanner at all. PATCH /api/items/:id/tracks only validates that each submitted ino is already in books.audioFiles, so it also doesn't update the values. Deleting and re-adding the item, the Plex dance, does fix it, but it also clears mediaProgresses, so no bueno.

I ended up repairing my database directly in the end, which is not... the ideal solution.

The reason I think this is worth fixing rather than filing under a "stop using Drivepool" tag is that the codebase has already accepted that inodes can change. compareUpdateLibraryFile updates libraryFile.ino, and checkLibraryItemData falls back to matching a library file by ino when the path lookup misses. Both of those exist because inode values can move. The copy on the book seems to be just a gap in that logic.

What did you expect to happen?

A scan reconciles the ino and file metadata on the book against the library file at the same path, so an affected item recovers instead of returning 404 indefinitely.

Steps to reproduce the issue

I haven't cleanly figured out how to repro this, so these steps force the same end state.

1. Add a book to a library and let it scan normally.
2. Stop the server. Change the ino recorded on the book so it no longer matches the file on disk, leaving libraryFiles alone:

   UPDATE books SET audioFiles = json_replace(audioFiles, '$[0].ino', '999999')
   WHERE id = (SELECT mediaId FROM libraryItems WHERE id = '<library item id>');

3. Start the server and run POST /api/items/<id>/scan. It returns {"result":"UPTODATE"} and the ino is still 999999.
4. GET /api/items/<id>/file/999999/download returns 404 and a mobile app reports the file as missing. Streaming should still work.

Scanning again, forced or not, does not move it back.

Audiobookshelf version

2.36.0

How are you running audiobookshelf?

Docker

What OS is your Audiobookshelf server hosted from?

Windows

If the issue is being seen in the UI, what browsers are you seeing the problem on?

None

Logs

Additional Notes

I am on edge at revision e8ed987 rather than the 2.36.0 release, but none of server/scanner/BookScanner.js, server/scanner/LibraryItemScanData.js, server/objects/files/AudioFile.js or server/controllers/LibraryItemController.js has changed between the v2.36.0 tag and master, so the code quoted above is the same.

My library sits on a Windows drive bind mounted into the container, and the inode numbers it reports changed at some point after the initial scan. The old values were a dense sequential range, which is a synthesized counter rather than any filesystem's real identifiers; the current ones are the NTFS 64-bit file reference. So the layer serving the bind mount changed how it numbers files, and that broke my inodes.

As near as I can tell, this is not specific to Windows. mergerfs computes inode values itself, offers several algorithms, and its own documentation notes that "Most software does not care what the values are but those that do often break if a value changes unexpectedly." Changing that setting renumbers everything. CIFS mounts without server-provided inodes have the same property, which is the underlying condition in issue #2509 here. Real block-level RAID is not affected, since it presents one filesystem with stable inodes.

While reading #2509 I noticed this has already been half-discussed here. nichwall asked whether "the inode should be ignored if the paths match", and specifically whether "if the file path matches and the modification time matches, could the inode just be updated to the new value without probing everything", and advplyr's answer was "This is correct and could be an area of improvement." That is close to what I ended up writing, down to not re-probing, so I think I am picking up a loose thread rather than proposing something new. The download 404 is just the symptom that sent me looking for it.

I have no idea how often this actually happens to people. Searching this repo and the app repo for "file not found" and "download failed" turned up nothing that looks like it. It is quiet when it does happen, though: streaming keeps working and the UI looks healthy, and only downloads break.

PodcastScanner has the same gate shape around its episode handling. I have no podcasts so I could not test it.

I have a fix and tests ready and will open a PR shortly.