Crash creating thumbnails

[zebrapurring]

First Check

• This is not a feature request
• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the documentation, with the integrated search.
• I already read the docs and didn't find an answer.
• I can replicate the issue inside the Demo install.

Homebox Version

v0.21.0

What is the issue you are experiencing?

After upgrading from v0.19.0, when I try to create the thumbnails I see the following error:

INF ../share/homebox/backend/internal/web/mid/logger.go:36 > request received method=POST path=/api/v1/actions/create-missing-thumbnails rid=homebox/***

 panic: ent: check existence: context canceled

 -> github.com/sysadminsmedia/homebox/backend/internal/data/ent.(*AttachmentQuery).ExistX
 ->   /usr/local/share/homebox/backend/internal/data/ent/attachment_query.go:285

    github.com/sysadminsmedia/homebox/backend/internal/data/repo.(*AttachmentRepo).CreateMissingThumbnails
      /usr/local/share/homebox/backend/internal/data/repo/repo_item_attachments.go:727
    main.(*app).mountRoutes.func1.(*V1Controller).HandleCreateMissingThumbnails.actionHandlerFactory.73
      /usr/local/share/homebox/backend/app/api/handlers/v1/v1_ctrl_actions.go:23
    github.com/hay-kot/httpkit/errchain.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/github.com/hay-kot/httpkit@v0.0.11/errchain/errchain.go:18
    main.(*app).mountRoutes.func1.(*app).mwRoles.5.1
      /usr/local/share/homebox/backend/app/api/middleware.go:69
    github.com/hay-kot/httpkit/errchain.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/github.com/hay-kot/httpkit@v0.0.11/errchain/errchain.go:18
    main.(*app).mwAuthToken-fm.(*app).mwAuthToken.func1
      /usr/local/share/homebox/backend/app/api/middleware.go:152
    github.com/hay-kot/httpkit/errchain.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/github.com/hay-kot/httpkit@v0.0.11/errchain/errchain.go:18
    main.run.Errors.func7.1
      /usr/local/share/homebox/backend/internal/web/mid/errors.go:22
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5.(*Mux).routeHTTP
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/mux.go:478
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5.(*Mux).ServeHTTP
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/mux.go:73
    github.com/go-chi/chi/v5.(*Mux).Mount.func1
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/mux.go:325
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5.(*Mux).routeHTTP
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/mux.go:478
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5/middleware.StripSlashes.func1
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/middleware/strip.go:31
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5/middleware.Recoverer.func1
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/middleware/recoverer.go:45
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    main.run.Logger.func6.1
      /usr/local/share/homebox/backend/internal/web/mid/logger.go:39
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5/middleware.RealIP.func1
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/middleware/realip.go:36
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5/middleware.RequestID.func1
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/middleware/request_id.go:76
    net/http.HandlerFunc.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2294
    github.com/go-chi/chi/v5.(*Mux).ServeHTTP
      /root/go/pkg/mod/github.com/go-chi/chi/v5@v5.2.2/mux.go:90
    net/http.serverHandler.ServeHTTP
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:3301
    net/http.(*conn).serve
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:2102
    created by net/http.(*Server).Serve in goroutine 68
      /root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.freebsd-amd64/src/net/http/server.go:3454

How can the maintainer reproduce the issue?

1. Upgrade installation from v0.19.0 with multiple attachments
2. Click "Create Thumbnails" button in "Tools"
3. The error above appears in the app logs

Deployment

Other

Database Type

SQLite

OS Architechture

x86_64 (AMD, Intel)

Deployment Details

Backend built with:

cd backend
COMMIT="$(git rev-parse HEAD)"
BUILD_TIME="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
VERSION="$(git describe --tags --abbrev=0)"
go build -o ../build/backend -ldflags "-s -w -X main.commit=${COMMIT} -X main.buildTime=${BUILD_TIME} -X main.version=${VERSION}" ./app/api

[tankerkiller125]

We currently provide FreeBSD releases: homebox_Freebsd_x86_64.tar.gz

You'll note that you need the nodynamic tag for FreeBSD builds to work properly (as per our GoReleaser infrastructure). This tells the underlying decoders/encoders to use the WebAssembly based versions instead of trying to use the C library versions directly. There's an extremely minor performance impact doing this, but very, very minor generally speaking. Also note that our builds turn CGO off.

[zebrapurring]

We currently provide FreeBSD releases: homebox_Freebsd_x86_64.tar.gz

You'll note that you need the nodynamic tag for FreeBSD builds to work properly (as per our GoReleaser infrastructure). This tells the underlying decoders/encoders to use the WebAssembly based versions instead of trying to use the C library versions directly. There's an extremely minor performance impact doing this, but very, very minor generally speaking. Also note that our builds turn CGO off.

Thanks for the hints! However I tried using the released binary and I can still reproduce the issue.

[zebrapurring]

I was able to reproduce it locally, from what I can see there is a deadlock in this line:

homebox/backend/internal/data/repo/repo_item_attachments.go

Line 864 in 397a1c6

group, err := r.db.Group.Get(ctx, groupId)

But I can't figure out why it is hanging.

Here are my settings:

HBOX_STORAGE_CONN_STRING="file:."
HBOX_DATABASE_SQLITE_PATH="./homebox.db?_pragma=busy_timeout=2000&_pragma=journal_mode=WAL&_fk=1"

[zebrapurring]

I think I found the issue; when POST /api/v1/actions/create-missing-thumbnails is received the CreateMissingThumbnails function triggers the thumbnail generation asynchronously with a pubsub topic, which is executed inside a transaction. However, after publishing to the topic function CreateMissingThumbnails starts querying the database in a loop. Since the SQLite connection pool is limited to 1:

homebox/backend/app/api/main.go

Line 143 in 397a1c6

db.SetMaxOpenConns(1)

This creates a deadlock because the backend is single threaded. This is what I think is happening:

1. CreateMissingThumbnails publishes to thumbnails topic
2. CreateMissingThumbnails runs async query and yields control
3. Subscription create-thumbnails-subscription is triggered and starts running
4. CreateThumbnail starts database transaction and returns control to CreateMissingThumbnails
5. CreateMissingThumbnails runs another query, which blocks the DB (since there is a running txn) and returns control to processThumbnailFromImage
6. processThumbnailFromImage runs a new query and it blocks

The solution from my point of view would be moving the block that signals messages as created to the create-thumbnails-subscription handler function. This way we could also check if an attachment has been created successfully, which is not verified with the current version.

In any case, I think we should also increase the maximum number of driver connections for SQLite to prevent other similar deadlocks.

I will try to prepare a PR implementing a fix and post it here.

[zebrapurring]

Seems the HTTP API handlers and the recurring runner tasks anyways do not share the same database connection, so the only solution I could find was to increase the maximum number of connections. See #1053 for a fix.

[tankerkiller125]

@zebrapurring given that the PR that introduced the 1 connection limit was after the last release (if I'm remembering the order correctly) I would find it strange if that was a root cause (at least for the pre-built binary).

[zebrapurring]

@tankerkiller125 my last tests were run on the latest mainline. I was able to reproduce it once with the prebuilt binary, but maybe that was because of failures processing non-image files that were not properly closing the transaction (c9d168c).

[tankerkiller125]

@zebrapurring The latest main branch reverted the connection limit, so you can try again. I should note however they we don't have great FreeBSD knowledge ourselves, so our troubleshooting on this one might be somewhat limited.

[zebrapurring]

@tankerkiller125 thanks for the update. I don't think the crash was related to FreeBSD (since I was also abale to reproduce it on macOS), rather the database settings and function CreateThumbnail returning without properly closing the database transaction.
I quickly tested the latest mainline and cannot reproduce the issue anymore.