What happened?
Downloading a book from a mobile app fails with "file not found" while streaming the same book works. The file is on disk, unchanged, and does play when accessed manually.
Audiobookshelf stores each audio file's inode twice, once in libraryItems.libraryFiles[].ino and then again in books.audioFiles[].ino. Remote clients download with GET /api/items/:id/file/:ino/download pulling the ino from media.audioFiles, and the handler compares that against libraryFiles. When the two disagree, the lookup misses and the download 404s.
On my server, annoyingly, they disagree for every audio file that came in during the initial library scan:
broken item, same relPath in both places
books.audioFiles[0].ino 14395
libraryItems.libraryFiles[].ino 281474976727661
stat on disk 281474976727661
working item, added months later
all three 281474976763149
GET /api/items/<broken>/file/14395/download -> 404
GET /api/items/<working>/file/281474976763149/download -> 206
I cannot tell you how the two got out of step, I have genuinely no idea how that could have happened. I thought it was a drivepool error, but it doesn't seem like it. compareUpdateLibraryFile does update libraryFile.ino when it changes and marks the file modified, which should carry the new value through to the book, so a scan that runs to completion ought to keep them together, but I've run multiple full scans of the library since then, and it was never caught. The two rows are saved separately, so if a scan died between the two saves it could cause this state, but I have no log proving that is what happened and I can't reproduce it consistently on an unmodified server.
I can show that once they have diverged, for whatever reason, no amount of full scanning will bring them back. You have to do the Plex dance to get it to work at scale.
POST /api/items/<broken>/scan -> {"result":"UPTODATE"}
The audio file block in BookScanner.rescanExistingBookLibraryItem sits behind one condition:
let hasMediaChanges = libraryItemData.hasAudioFileChanges || libraryItemData.audioLibraryFiles.length !== media.audioFiles.length
hasAudioFileChanges only updates if something changes between the last two scans. libraryFiles already holds the correct value, so every additional scan sees no changes. A library scan with force reaches rescanLibraryItemMedia but that condition still fails to update the ino. An ordinary scan of an unchanged item does not use BookScanner at all. PATCH /api/items/:id/tracks only validates that each submitted ino is already in books.audioFiles, so it also doesn't update the values. Deleting and re-adding the item, the Plex dance, does fix it, but it also clears mediaProgresses, so no bueno.
I ended up repairing my database directly in the end, which is not... the ideal solution.
The reason I think this is worth fixing rather than filing under a "stop using Drivepool" tag is that the codebase has already accepted that inodes can change. compareUpdateLibraryFile updates libraryFile.ino, and checkLibraryItemData falls back to matching a library file by ino when the path lookup misses. Both of those exist because inode values can move. The copy on the book seems to be just a gap in that logic.
What did you expect to happen?
A scan reconciles the ino and file metadata on the book against the library file at the same path, so an affected item recovers instead of returning 404 indefinitely.
Steps to reproduce the issue
I haven't cleanly figured out how to repro this, so these steps force the same end state.
- Add a book to a library and let it scan normally.
- Stop the server. Change the ino recorded on the book so it no longer matches the file on disk, leaving
libraryFiles alone:
UPDATE books SET audioFiles = json_replace(audioFiles, '$[0].ino', '999999')
WHERE id = (SELECT mediaId FROM libraryItems WHERE id = '<library item id>');
- Start the server and run
POST /api/items/<id>/scan. It returns {"result":"UPTODATE"} and the ino is still 999999.
GET /api/items/<id>/file/999999/download returns 404 and a mobile app reports the file as missing. Streaming should still work.
Scanning again, forced or not, does not move it back.
Audiobookshelf version
2.36.0
How are you running audiobookshelf?
Docker
What OS is your Audiobookshelf server hosted from?
Windows
If the issue is being seen in the UI, what browsers are you seeing the problem on?
None
Logs
Additional Notes
I am on edge at revision e8ed987 rather than the 2.36.0 release, but none of server/scanner/BookScanner.js, server/scanner/LibraryItemScanData.js, server/objects/files/AudioFile.js or server/controllers/LibraryItemController.js has changed between the v2.36.0 tag and master, so the code quoted above is the same.
My library sits on a Windows drive bind mounted into the container, and the inode numbers it reports changed at some point after the initial scan. The old values were a dense sequential range, which is a synthesized counter rather than any filesystem's real identifiers; the current ones are the NTFS 64-bit file reference. So the layer serving the bind mount changed how it numbers files, and that broke my inodes.
As near as I can tell, this is not specific to Windows. mergerfs computes inode values itself, offers several algorithms, and its own documentation notes that "Most software does not care what the values are but those that do often break if a value changes unexpectedly." Changing that setting renumbers everything. CIFS mounts without server-provided inodes have the same property, which is the underlying condition in issue #2509 here. Real block-level RAID is not affected, since it presents one filesystem with stable inodes.
While reading #2509 I noticed this has already been half-discussed here. nichwall asked whether "the inode should be ignored if the paths match", and specifically whether "if the file path matches and the modification time matches, could the inode just be updated to the new value without probing everything", and advplyr's answer was "This is correct and could be an area of improvement." That is close to what I ended up writing, down to not re-probing, so I think I am picking up a loose thread rather than proposing something new. The download 404 is just the symptom that sent me looking for it.
I have no idea how often this actually happens to people. Searching this repo and the app repo for "file not found" and "download failed" turned up nothing that looks like it. It is quiet when it does happen, though: streaming keeps working and the UI looks healthy, and only downloads break.
PodcastScanner has the same gate shape around its episode handling. I have no podcasts so I could not test it.
I have a fix and tests ready and will open a PR shortly.
What happened?
Downloading a book from a mobile app fails with "file not found" while streaming the same book works. The file is on disk, unchanged, and does play when accessed manually.
Audiobookshelf stores each audio file's inode twice, once in
libraryItems.libraryFiles[].inoand then again inbooks.audioFiles[].ino. Remote clients download withGET /api/items/:id/file/:ino/downloadpulling the ino frommedia.audioFiles, and the handler compares that againstlibraryFiles. When the two disagree, the lookup misses and the download 404s.On my server, annoyingly, they disagree for every audio file that came in during the initial library scan:
I cannot tell you how the two got out of step, I have genuinely no idea how that could have happened. I thought it was a drivepool error, but it doesn't seem like it.
compareUpdateLibraryFiledoes updatelibraryFile.inowhen it changes and marks the file modified, which should carry the new value through to the book, so a scan that runs to completion ought to keep them together, but I've run multiple full scans of the library since then, and it was never caught. The two rows are saved separately, so if a scan died between the two saves it could cause this state, but I have no log proving that is what happened and I can't reproduce it consistently on an unmodified server.I can show that once they have diverged, for whatever reason, no amount of full scanning will bring them back. You have to do the Plex dance to get it to work at scale.
The audio file block in
BookScanner.rescanExistingBookLibraryItemsits behind one condition:hasAudioFileChangesonly updates if something changes between the last two scans.libraryFilesalready holds the correct value, so every additional scan sees no changes. A library scan withforcereachesrescanLibraryItemMediabut that condition still fails to update the ino. An ordinary scan of an unchanged item does not useBookScannerat all.PATCH /api/items/:id/tracksonly validates that each submitted ino is already inbooks.audioFiles, so it also doesn't update the values. Deleting and re-adding the item, the Plex dance, does fix it, but it also clearsmediaProgresses, so no bueno.I ended up repairing my database directly in the end, which is not... the ideal solution.
The reason I think this is worth fixing rather than filing under a "stop using Drivepool" tag is that the codebase has already accepted that inodes can change.
compareUpdateLibraryFileupdateslibraryFile.ino, andcheckLibraryItemDatafalls back to matching a library file by ino when the path lookup misses. Both of those exist because inode values can move. The copy on the book seems to be just a gap in that logic.What did you expect to happen?
A scan reconciles the ino and file metadata on the book against the library file at the same path, so an affected item recovers instead of returning 404 indefinitely.
Steps to reproduce the issue
I haven't cleanly figured out how to repro this, so these steps force the same end state.
libraryFilesalone:POST /api/items/<id>/scan. It returns{"result":"UPTODATE"}and the ino is still999999.GET /api/items/<id>/file/999999/downloadreturns 404 and a mobile app reports the file as missing. Streaming should still work.Scanning again, forced or not, does not move it back.
Audiobookshelf version
2.36.0
How are you running audiobookshelf?
Docker
What OS is your Audiobookshelf server hosted from?
Windows
If the issue is being seen in the UI, what browsers are you seeing the problem on?
None
Logs
Additional Notes
I am on
edgeat revision e8ed987 rather than the 2.36.0 release, but none ofserver/scanner/BookScanner.js,server/scanner/LibraryItemScanData.js,server/objects/files/AudioFile.jsorserver/controllers/LibraryItemController.jshas changed between the v2.36.0 tag and master, so the code quoted above is the same.My library sits on a Windows drive bind mounted into the container, and the inode numbers it reports changed at some point after the initial scan. The old values were a dense sequential range, which is a synthesized counter rather than any filesystem's real identifiers; the current ones are the NTFS 64-bit file reference. So the layer serving the bind mount changed how it numbers files, and that broke my inodes.
As near as I can tell, this is not specific to Windows. mergerfs computes inode values itself, offers several algorithms, and its own documentation notes that "Most software does not care what the values are but those that do often break if a value changes unexpectedly." Changing that setting renumbers everything. CIFS mounts without server-provided inodes have the same property, which is the underlying condition in issue #2509 here. Real block-level RAID is not affected, since it presents one filesystem with stable inodes.
While reading #2509 I noticed this has already been half-discussed here. nichwall asked whether "the inode should be ignored if the paths match", and specifically whether "if the file path matches and the modification time matches, could the inode just be updated to the new value without probing everything", and advplyr's answer was "This is correct and could be an area of improvement." That is close to what I ended up writing, down to not re-probing, so I think I am picking up a loose thread rather than proposing something new. The download 404 is just the symptom that sent me looking for it.
I have no idea how often this actually happens to people. Searching this repo and the app repo for "file not found" and "download failed" turned up nothing that looks like it. It is quiet when it does happen, though: streaming keeps working and the UI looks healthy, and only downloads break.
PodcastScannerhas the same gate shape around its episode handling. I have no podcasts so I could not test it.I have a fix and tests ready and will open a PR shortly.