Skip to content

Add lightweight web UI for monitoring, activity history & runtime control#337

Open
lolimmlost wants to merge 46 commits into
ManiMatter:devfrom
lolimmlost:feat/web-ui
Open

Add lightweight web UI for monitoring, activity history & runtime control#337
lolimmlost wants to merge 46 commits into
ManiMatter:devfrom
lolimmlost:feat/web-ui

Conversation

@lolimmlost

Copy link
Copy Markdown
Collaborator

Summary

Decluttarr currently has zero visibility into what it's doing — all config is YAML, all output is logs. This PR adds a lightweight web UI for monitoring, activity history, and runtime control without changing the existing daemon behavior.

  • Dashboard — real-time queue view across all arr instances, instance status cards, live activity feed, "Run Now" button
  • Activity Log — searchable, filterable, paginated history of every action (flags, removals, recoveries, strikes) stored in SQLite
  • Settings Editor — toggle test_run, enable/disable jobs, adjust max_strikes/min_speed at runtime without editing YAML or restarting
  • Download Protection — protect individual downloads from removal via the UI (supplements the qBit "Keep" tag)
  • REST API — full JSON API with auto-generated OpenAPI docs at /api/docs
  • SSE Live Updates — server-sent events push changes to the browser in real time

Tech Choices

Component Choice Why
Web framework FastAPI Async-native (shares existing asyncio loop), lightweight, built-in OpenAPI
Frontend Jinja2 + HTMX + Alpine.js No build step, no Node tooling in a Python project
Styling Pico CSS (dark theme) Classless CSS, minimal custom styles needed
Persistence SQLite via aiosqlite Zero config, file-based, auto-creates on first run
Real-time Server-Sent Events Simpler than WebSockets, unidirectional, HTMX-compatible

Architecture

The web server runs as a sibling asyncio task alongside the existing main loop — both share the same event loop and process memory. An EventBus class decouples the job system from the UI: jobs emit events at decision points, the web layer (ActivityRecorder + SSE) consumes them. When web is disabled, a NoOpEventBus is used with zero overhead.

Job System → EventBus → ActivityRecorder (writes SQLite)
                      → SSE endpoint (pushes to browser)

Browser → FastAPI API → reads Tracker state (queue/strikes)
                      → reads/writes SQLite (activity, config, protected)
                      → mutates Settings object (runtime config)

Database Schema (SQLite)

Three tables: activity_log (action history), protected_downloads (UI-managed protection), config_overrides (runtime config layered on top of YAML). Auto-created at ./data/decluttarr.db.

API Endpoints

Method Path Purpose
GET /api/status Uptime, test_run state, instance count
GET /api/queue Current queue across all arr instances with strike info
GET /api/activity Paginated activity log with filters
GET /api/strikes Current strike data across all trackers
POST/DELETE /api/protected/{id} Protect/unprotect a download
GET/PATCH /api/config Read/update runtime config
POST /api/config/test-run Toggle test_run on/off
POST /api/config/reload Reset overrides to YAML defaults
GET /api/events SSE stream for real-time updates
POST /api/trigger Manually trigger a job cycle

Configuration

Zero new required config. Defaults to enabled on port 9999.

# config.yaml (optional)
web:
  enabled: true    # or WEB_ENABLED=false to disable
  host: "0.0.0.0"
  port: 9999

Migration / Backward Compatibility

  • Defaults to enabled but works with zero config — existing YAML configs unaffected
  • No new required env vars — all web settings have sensible defaults
  • Event bus is no-op when web is disabled — zero overhead on existing behavior
  • Database auto-creates on first run
  • All 192 existing tests pass unchanged

New Dependencies

fastapi==0.115.6
uvicorn[standard]==0.34.0
aiosqlite==0.20.0
jinja2==3.1.5
python-multipart==0.0.20

Files Changed

New (15 files in src/web/): events.py, database.py, app.py, routes.py, config_manager.py, templates (base, dashboard, activity, settings, 4 partials), static/style.css

Modified (11 files): main.py, job_manager.py, removal_job.py, removal_handler.py, strikes_handler.py, _general.py, _user_config.py, _instances.py, Dockerfile, requirements.txt, config_example.yaml

Screenshots

The UI uses Pico CSS dark theme with color-coded badges for arr instances (Sonarr=blue, Radarr=yellow, etc.), action types (removed=red, recovered=green, flagged=amber), and strike counts.

Test Plan

  • pytest tests/ — all 192 existing tests pass
  • Web UI loads at http://localhost:9999
  • Job loop still runs on timer (verified via logs)
  • Queue table shows downloads with strike/protection status
  • Protect/unprotect buttons work and survive next cycle
  • test_run toggle via settings page takes immediate effect
  • "Run Now" button triggers early cycle
  • Activity log records and displays actions
  • Docker build succeeds with EXPOSE 9999
  • Verify with WEB_ENABLED=false that web is fully disabled
  • Test with multiple concurrent SSE clients

🤖 Generated with Claude Code

@Rubilmax

Copy link
Copy Markdown

Can we get this reviewed and merged please?

@lolimmlost

Copy link
Copy Markdown
Collaborator Author

Can we get this reviewed and merged please?

I appreciate your enthusiasm but definitively needs testing as I'm getting webui errors after a weeklong usage. I'll review the code once again this weekend.

@Rubilmax

Copy link
Copy Markdown

Amazing, thanks 🙏

@lolimmlost

Copy link
Copy Markdown
Collaborator Author

I'm attempting this fix for the crashing.

@lolimmlost

Copy link
Copy Markdown
Collaborator Author

Pushed a fix for a crash that was happening after ~1 week of uptime.

Root cause: When Sonarr/Radarr timed out (read timeout=15s), the unhandled exception propagated up through asyncio.gather(main_task, web_task), which cancelled the web server task too — killing the entire app.

Fix (commit 25f3e2f):

  • Wrapped per-instance job runs and download client jobs in try/except so timeouts log an error and continue to the next cycle
  • Added main_with_restart() wrapper so even unexpected failures auto-recover after 30s while the web UI stays up independently

Verified running 24hrs+ on production with multiple Sonarr/Radarr timeouts — all recovered cleanly on the next cycle, no crashes.

@lolimmlost

Copy link
Copy Markdown
Collaborator Author

I have tested the new fixes with 100% uptime after 48 hours. Ill be working on ui improvements; log pruning and api cache control.
Is pasword protections something we'd like for this but i assume the end user can secure it however they'd like. (cf access)

@ManiMatter

Copy link
Copy Markdown
Owner

hi, I am truly sorry I haven't looked into your PR in such a long time. I do appreciate very much that you took the time to contribute.

Unfortunately, I don't have the time to look into it still.
To overcome me being the bottleneck, I am looking to open this repo up to other people who help maintain it, and contributors can review each others code / merge.

Would you be willing to act as a formal contributor? If yes, I will add you, and if I find others (from open PRs), hopefully you can review each others PR and they can be merged.

Thanks for letting me know, and apologies again for my radio silence.

@lolimmlost

lolimmlost commented Apr 18, 2026 via email

Copy link
Copy Markdown
Collaborator Author

@ManiMatter

Copy link
Copy Markdown
Owner

Hey @ManiMatter. Thanks you for your honesty and I appreciate you wanting this project to continue. I personally would like to be a contributor. However I would need direction / goal in mind that we can work towards together. I am open to discussing what the future of the project may look like. Thanks for this opportunity.

On Sat, Apr 18, 2026 at 4:11 AM ManiMatter @.> wrote: ManiMatter left a comment (ManiMatter/decluttarr#337) <#337?email_source=notifications&email_token=ADBC4Q4G7LPLSTC5DZZ4AZL4WNPHLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRXGM2TCMJVG44KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2LK4DSL5RW63LNMVXHIX3POBSW4X3DNRUWG2Y#issuecomment-4273511578> hi, I am truly sorry I haven't looked into your PR in such a long time. I do appreciate very much that you took the time to contribute. Unfortunately, I don't have the time to look into it still. To overcome me being the bottleneck, I am looking to open this repo up to other people who help maintain it, and contributors can review each others code / merge. Would you be willing to act as a formal contributor? If yes, I will add you, and if I find others (from open PRs), hopefully you can review each others PR and they can be merged. Thanks for letting me know, and apologies again for my radio silence. — Reply to this email directly, view it on GitHub <#337?email_source=notifications&email_token=ADBC4Q4G7LPLSTC5DZZ4AZL4WNPHLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRXGM2TCMJVG44KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2LK4DSL5RW63LNMVXHIX3POBSW4X3DNRUWG2Y#issuecomment-4273511578>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADBC4Q3BUVJF6NFQEKXD44T4WNPHLAVCNFSM6AAAAACWSAOXR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DENZTGUYTCNJXHA . You are receiving this because you authored the thread.Message ID: @.>

hi @lolimmlost - Awesome, I am so glad you raise your hand to become a contributor.
I have just added you to the contributor list.

I also started a "Discussion", suggest we take the exchange there on what to focus on next / where to bring the tool from here. #345

@ManiMatter

ManiMatter commented Apr 19, 2026

Copy link
Copy Markdown
Owner

First of all - looks awesome. I think this is a massive improvement for the tool!
Some thoughts on this PR:

  1. Could you update the Readme to point out that there is a UI now, how to use it etc?
  2. This version will create an initial database; how do we deal with a future situation where changes to existing tables are needed? Would it make sense to already now use "alembic" (potentially in conjunction with sqlalchemy?).
    ̶3̶)̶ ̶/̶d̶a̶t̶a̶ ̶s̶h̶o̶u̶l̶d̶ ̶b̶e̶ ̶e̶x̶c̶l̶u̶d̶e̶d̶ ̶v̶i̶a̶ ̶.̶g̶i̶t̶i̶g̶n̶o̶r̶e̶/̶.̶d̶o̶c̶k̶e̶r̶i̶g̶n̶o̶r̶e̶ (done. see commits above)
    ̶4̶)̶ ̶a̶d̶d̶ ̶s̶u̶p̶p̶o̶r̶t̶ ̶t̶o̶ ̶r̶u̶n̶ ̶t̶h̶i̶s̶ ̶b̶e̶h̶i̶n̶d̶ ̶p̶r̶o̶x̶y̶ ̶(̶f̶o̶r̶ ̶i̶n̶s̶t̶a̶n̶c̶e̶ ̶w̶h̶e̶n̶ ̶o̶n̶ ̶c̶o̶d̶e̶s̶e̶r̶v̶e̶r̶)̶ (done. see commits above)
  3. the "web" variables are currently stored in the "general" section of the settings class. I don't think that's right; they should be in "web" - Could you please amend? the proxy variable (4) I added above I also put in general since the web-section doesn't exist yet, please also move it along.
  4. Do you think it could be possible to edit the instances also via the UI? The nested yaml structure to support the different instances is something that comes up in issues again and again. If this could be done more easily via UI, this would be a big win.
  5. Also, managing the settings per the different jobs would be great; which would eliminate the need for users to fight yaml as they could configure everything via UI

@lolimmlost

Copy link
Copy Markdown
Collaborator Author

Hey @ManiMatter, thanks for the detailed feedback! I've addressed your requests in the latest commits:

Done in this PR:

  • Container not starting #1 README — Added a Web UI section covering features, configuration (YAML + env vars), Docker port mapping, and how to disable it (d52a011)
  • Container does not restart after an error #2 Database migration strategy — Added a lightweight schema versioning system with a schema_version table and numbered migrations. Avoids the Alembic dependency (overkill for 3-table SQLite) while making future schema changes safe for existing databases (3685932)
  • Deleting automatically old DEV packages #5 Web settings in own section — Extracted web_enabled, web_host, web_port, and proxy_prefix from General into a dedicated Web settings class in src/settings/_web.py, following the same pattern as _jobs.py, _download_clients.py, etc. Updated all consumers (a05a812)

Already done (by you):

#6 (Instance editing via UI) and #7 (Full job config via UI):
The settings page already supports toggling jobs and adjusting max_strikes/min_speed at runtime (#7 partial). However, full instance editing (#6) and complete job config management (#7) would be significant additions — instance config involves nested YAML structures with API keys and multiple arr types.

Would it make sense to merge this PR as-is and open separate PRs for #6 and #7 as follow-up features? Happy to take those on as a contributor.

Also pushed a few additional fixes while testing:

  • Fixed XSS vulnerability in queue table onclick handlers (switched to data-* attributes with event delegation)
  • Fixed wait_and_exit() regression when web UI is disabled (non-web path now also gets restart-on-failure)
  • Added SRI integrity hashes to CDN-loaded assets (Pico CSS, HTMX, Alpine.js)
  • Fixed uvicorn root_path=None causing 400 Bad Request

@ManiMatter

ManiMatter commented Apr 21, 2026

Copy link
Copy Markdown
Owner

Thank you, @lolimmlost for the additional changes.
I think your proposal makes a lot of sense to keep 6) and 7) separate and this can be merged as first version.

Before merging though I think it would be good if somebody could review this code in more detail, as it is a relatively big addition.

Time wise I won‘t be able to do it myself. I hope somebody volunteers as additional maintainer to you & me and reviews/merges this.

@lolimmlost As you look into 6) and 7), would you be willing to review #311? The author there introduced per-instance overrides, which, if merged, would play into 7) (ie. full config via (UI) thus I see them related.

@ManiMatter ManiMatter changed the title feat: Add lightweight web UI for monitoring, activity history & runtime control [NEEDS CODE REVIEWER] Add lightweight web UI for monitoring, activity history & runtime control Apr 21, 2026
@ManiMatter

This comment was marked as off-topic.

@lolimmlost

This comment was marked as off-topic.

@ManiMatter

This comment was marked as off-topic.

@lolimmlost

lolimmlost commented Apr 28, 2026 via email

Copy link
Copy Markdown
Collaborator Author

@ManiMatter

Copy link
Copy Markdown
Owner

Cool. feel free to take on anything you want, for example open PRs, issues, or suggest new changes as you see fit.
There are definitely items where this tool would benefit from a code overhaul (it's my first python project ever...), thus feel free to take a go at anything you'd like and feel fully free to take any decisions (and tag me in tickets if you want a second opinion) :)

Suggest we update #345 for wider discussions if needed.

@lolimmlost

Copy link
Copy Markdown
Collaborator Author

Hey @ManiMatter, did a self-review pass on this PR and pushed fixes for what I found. One thing I'd like your call on before merging — it came from your a840262 commit so I didn't want to touch it unilaterally.

The thing: in src/web/app.py:51 the proxy support builds:

root_path = f"/{proxy_prefix}/{port}" if proxy_prefix else ""

Embedding the listen port matches code-server's /proxy/<port>/... convention. But users behind nginx/Traefik/Caddy expect a clean prefix like /decluttarr without the port — current code generates a root_path they don't want.

Options:

  1. Keep as-is, document that proxy_prefix is code-server-specific
  2. Drop the port: root_path = f"/{proxy_prefix}", code-server users set proxy_prefix: "proxy/9999"
  3. Add a flag like code_server_mode: true to switch formats

Leaning toward 2 — cleaner, and code-server still works with one extra path segment in config. What do you prefer?

@ManiMatter

Copy link
Copy Markdown
Owner

Hey @lolimmlost, thanks for asking. I agree with your proposal to go for option 2, so that the same setting can be used for any proxy

@Dark3clipse Dark3clipse self-assigned this May 2, 2026
@Dark3clipse

Copy link
Copy Markdown
Collaborator

Hi @lolimmlost, I just joined this project as a maintainer to help out, and I'd like to spend some time to fully review your PR.

@lolimmlost

lolimmlost commented May 2, 2026 via email

Copy link
Copy Markdown
Collaborator Author

@Dark3clipse

Dark3clipse commented May 2, 2026

Copy link
Copy Markdown
Collaborator

So, I've spend my evening reviewing your new GUI. I have deployed your branch in my environment and reviewed the frontend design, usability, and deployment. I have yet to review the code itself, I will do that at a later time.

However, I'd already like to post my initial review. It focuses on the architecture, deployment and usability, and frontend. I've used ai solely to format the review nicely and help me with better sentence structure.

PR Review Feedback

Architectural

1. Optional External Database Support (Future Consideration)

It may be worth considering support for an external database such as PostgreSQL (not required for this PR, but valuable long-term).

Benefits:

  • Keeps decluttarr deployments stateless (no need for persistent volumes)
  • Allows users to leverage existing infrastructure (backups, monitoring, optimization)
  • Many users already running a full arr stack + Plex/Jellyfin likely have PostgreSQL available

2. Configurable SQLite Database Location

It would be useful to allow users to specify where the SQLite database file is stored.

Benefits:

  • Enables better separation of persistent storage (e.g., dedicated PVCs)

Alternative:

  • Clearly document which directory should be mounted/bound for persistence

3. Frontend Dependency on External CDNs

The web app currently uses a no-build frontend approach, but still depends on runtime external resources (e.g., unpkg.com, jsDelivr for docs).

Concerns:

  • Introduces external runtime dependencies
  • Causes issues in restricted or offline environments
  • Requires additional CSP configuration (e.g., script-src 'self' https://unpkg.com)
  • Adds security considerations

Recommendation:
Vendor the required JS files directly into the repository (e.g., /web/static/*.min.js).

Advantages:

  • No CDN dependency
  • Works in airgapped or restricted environments
  • No CSP adjustments required for reverse proxies
  • Keeps the current lightweight, no-build approach

4. CSP Compatibility & Inline Scripts

Testing with a stricter Content Security Policy revealed that the frontend relies on inline scripts.

To make this work, CSP would require:

'unsafe-inline'

This is not recommended: https://content-security-policy.com/unsafe-inline/

Additionally, there are indications of eval usage:

Uncaught EvalError: Evaluating a string as JavaScript violates the following Content Security Policy directive...
csp

Recommendations:

  • Move inline scripts into separate JS files
  • Avoid patterns requiring unsafe-inline or unsafe-eval

Note:
This may be related to the use of Alpine.js. If so, it’s worth evaluating whether:

  • Alpine usage can be adjusted
  • Or a simpler approach (e.g., HTMX-only) would reduce CSP friction

5. Support for Read-Only Configurations

It would be beneficial to continue supporting users who provide configuration as read-only (e.g., via infrastructure-as-code workflows).

Use case:

  • Users define config in Git and mount it into the container
  • Prefer not to mutate config at runtime

Suggestion:

  • Detect read-only config files or introduce a readonly mode
  • Disable or grey out UI controls that modify configuration

This still allows the UI to provide value for visibility and monitoring.


⚙️ Usage

1. Historical Runs Overview (Future Feature)

It would be useful to:

  • View a list of past runs
  • Click into a run to inspect detailed logs/tracing

2. Download Queue Integrations (Future Feature)

Enhancements could include clickable links to:

  • Open the corresponding *arr instance (highlighting the item)
  • Open the download client (optionally with qui support)

Frontend

1. Responsiveness / Layout Issues

The Download Queue section currently introduces a horizontal scrollbar (especially noticeable on vertical monitors).
scrollbar

Observations:

  • The “Status” column appears too wide
  • Layout does not scale well on smaller screens
  • All table views are affected

Suggestions:

  • Improve responsiveness so rows wrap instead of overflowing
  • On small screens (e.g., mobile), allow rows to grow vertically instead of forcing horizontal scroll
  • Ensure all list views respect .container padding

2. Spacing Improvements

Some additional vertical spacing would improve readability:

  • Between instance cards
  • Between the “Run Now” / toggle controls
  • Above the download queue

3. Pagination for Large Lists

Currently, large lists (e.g., Download Queue) render all items at once.

Issue:

  • Requires excessive scrolling (e.g., 700+ items before reaching “Recent Activity”)

Suggestion:

  • Introduce pagination (as already done on the Activity page)
  • Apply consistently across list views

4. Project Branding / Favicon

It might be a good time for us to introduce:

  • A simple project logo
  • A favicon for browser tabs

This improves usability when multiple tabs are open. @ManiMatter how do you feel about this?


5. Pager Styling

The current pager UI feels slightly cramped.

Suggestion:

  • Consider using display: flex for better spacing/alignment

(This improved the layout in local testing, but open to preference.)
flex


✅ Summary

Overall, I really enjoy this new addition to decluttarr, and I think it is very valuable. I think it is a great next step for the project, and I'd like to thank you for your contribution.

The main concerns in my review center around:

  • Eliminating external frontend dependencies (CDNs)
  • Improving CSP compatibility
  • Maintaining compatibility with common deployment patterns (stateless, read-only config)
  • Enhancing usability for larger datasets and smaller screens

Addressing these will significantly improve robustness and maintainability, as well as deliver an even snappier frontend to our users.

@ManiMatter

ManiMatter commented May 2, 2026

Copy link
Copy Markdown
Owner

Really appreciate you take the time to review this thoroughly.
One item that got my attention is this one:

1. Optional External Database Support (Future Consideration)

It may be worth considering support for an external database such as PostgreSQL (not required for this PR, but valuable long-term).

Benefits:

  • Keeps decluttarr deployments stateless (no need for persistent volumes)
  • Allows users to leverage existing infrastructure (backups, monitoring, optimization)
  • Many users already running a full arr stack + Plex/Jellyfin likely have PostgreSQL available

Could you please elaborate why a external DB would be handy? My worry is that adding PostGresDB would create an additional external dependency which a) might increase application/development complexity, b) might add setup complexity for users (for instance, when running standalone python script outside docker enviornment)

Re your point on "Many users already running a full _arr stack" - That definitely applies to me (using Plex, Radarr, etc), but I for instance do not use PostGres for any of these applications but rely on their built-in DBs. Until you suggested this addtion here, I wasn't even aware an external DB could be used (and haven't really understood the point yet tbh, but since the feature exists and you ask for it, I'm sure there is good reasons for it)

I am not an expert in DBs at all, thus pls don't get discouraged by my thoughts, keen to hear your perspectives.


on this point:

  1. Project Branding / Favicon
    It might be a good time for us to introduce:

A simple project logo
A favicon for browser tabs
This improves usability when multiple tabs are open.

Like the idea. Feel free to create any icon you think makes sense; we can always change it in the future for something else if anybody has strong feelings.

@Dark3clipse

Copy link
Copy Markdown
Collaborator

Ah I can elaborate on the benefits of an external DB, certainly.

SQlite benefits

Sqlite is my no means a bad default. It serves a good purpose:

  • zero configuration
  • no additional services required, making it quicker and simpler to deploy decluttarr for most users
  • Perfect for small, single-instance setups

My recommendation is not to replace sqlite, rather I think it should be the default db option.

I'm kind of a power user myself, since I run a homelab with kubernetes. And specifically for such use cases, the benefits of external database support in applications becomes visible. So please don't consider my suggestion a must :) But I think it is not a lot of effort to implement external db support (I'd like to contribute myself) especially given that a db will be a new component for the project.

External database support is beneficial when users run more advanced setups

  • large docker compose stacks
  • Kubernetes
  • full homelabs

Drawbacks of sqlite

1. Stateful storage requirement

  • SQLite requires a local file
  • This means:
    • Docker → bind mounts or named volumes
    • Kubernetes → PersistentVolumeClaims (PVCs)
  • This introduces:
    • Storage lifecycle management
    • Backup complexity
    • Migration concerns

For example, in my k8s setup, I'd need to create a PVC and configure snapshot and backup schedule for it separately, whereas I already have a good snapshot/backup configuration for my postgres instance.
image

2. Reduced portability

  • A container is no longer self-contained
  • Deployments now depend on external storage configuration
  • Moving workloads between nodes (Kubernetes) becomes more complex

Again, not a problem at all for most users, but homelab users will need extra effort.

3. Concurrency limitations

  • SQLite has limited support for concurrent writes
  • While this may not be an issue today, the introduction of a web UI + background jobs increases the chance of:
    • overlapping operations
    • locking contention

I don't think we will run into this given the scope of this project.

4. Scaling constraints

  • SQLite does not scale beyond a single instance
  • Running multiple replicas (even accidentally) can lead to:
    • database corruption risks
    • undefined behavior

You can see for example Jellyfin is working towards migrating from a local db to support for external db:
https://jellyfin.org/posts/jellyfin-release-10.11.0/

Which will enable them to scale. In practice, this would enable users to run multiple instances of jellyfin with failover when one of them becomes unstable to bring more stability towards users of their Jellyfin deployment.

For decluttarr this is not relevant, but it serves of an example why external databases can scale better.

Benefits of External Databases (e.g., PostgreSQL)

Supporting an external database unlocks several operational advantages:

1. Truly stateless application containers

  • No local storage required
  • Containers can be:
    • ephemeral
    • easily rescheduled
    • horizontally scalable (if ever needed)

This aligns well with Kubernetes best practices.


2. Easier backups and disaster recovery

  • Many users already have:
    • automated backups
    • replication
    • monitoring
  • Decluttarr can integrate into existing database infrastructure instead of introducing a new backup surface

3. Reuse of existing infrastructure

  • Users running:
    • *arr stack
    • Plex / Jellyfin
    • other services

often already operate PostgreSQL or MariaDB instances.

Allowing reuse:

  • reduces duplication
  • simplifies operations
  • improves consistency across services

4. Better support for multi-instance or future growth

  • Even if not a current goal, external DBs enable:
    • multiple workers
    • separation of UI and worker components
    • safer concurrent access patterns

5. Improved reliability in orchestrated environments

  • Kubernetes scheduling becomes simpler:
    • no need to co-locate pods with storage
    • no risk of losing local state on rescheduling
  • Works better with:
    • rolling updates
    • scaling events
    • node failures

Docker Compose Perspective

Even in Docker Compose setups, external DBs can be beneficial:

  • Many users already define a shared postgres service
  • Centralized backups are easier than managing multiple SQLite files
  • Cleaner separation of concerns between:
    • application
    • data layer

I've had in the past when I was using docker-compose myself problems with sqlite corruption in my bazarr instance, forcing me to start over. For docker-compose users, it would be easier to configure postgres and make sure that that is properly backed up, than to make sure each and every application that uses sqlite has proper backup.

That said, SQLite should remain a first-class default for simple setups.

Recommended Approach

What I would recommend is the following:

  • don't spend more attention to this topic for this PR
  • Keep SQLite as the default (zero-config experience)

After this has been merged, I'll start a new branch and experiment with external db support:

  • Add optional support for an external database (e.g., PostgreSQL)
  • Allow configuration via environment variables
  • Clearly document both modes:
    • “simple mode” (SQLite)
    • “advanced mode” (external DB)

@ManiMatter

Copy link
Copy Markdown
Owner

Really appreciate the detailed answer and explanation, I enjoyed the thoughtful read and learnt something today.
Great idea to park this for now and thank you for looking into this once this PR is merged.

lolimmlost and others added 28 commits May 24, 2026 21:20
Document the web UI feature including dashboard, activity log, settings
editor, download protection, REST API, and SSE live updates. Covers
configuration (YAML and env vars), Docker port mapping, and how to
disable the web UI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace raw SCHEMA string with a MIGRATIONS list and a schema_version
table. Database.init() now runs only pending migrations, making future
schema changes safe for existing databases.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace inline onclick handlers with data-* attributes and event
delegation. download_id and arr_name were rendered unescaped in JS
string context — a crafted value could break out of the string.
Event delegation also works correctly with HTMX-swapped content.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When web_enabled=False, sys.exit(1) from wait_and_exit() propagated
directly and killed the process. Now both paths use main_with_restart()
so unreachable services trigger a 30s retry instead of a hard crash.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add integrity and crossorigin attributes to Pico CSS, HTMX, and
Alpine.js script/link tags to prevent supply-chain tampering via
compromised CDNs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
uvicorn.Config expects root_path to be a string (default ''), not None.
Passing None caused all HTTP requests to return 400 Bad Request.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous hash was generated without following the unpkg redirect,
resulting in a hash of the redirect page instead of the actual JS file.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The activity page fetched /api/activity directly without prefixing
rootPath, breaking the page when running behind a reverse proxy.
Every other fetch in the codebase already uses rootPath.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
api_protect compared content-type with strict equality, so a request
with the standard 'application/json; charset=utf-8' header fell through
and the body was silently dropped — protected_downloads ended up with
title='Unknown' and an empty arr_name. Use startswith() to accept the
charset suffix.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
_validate_config_key only checked the attribute portion of a 'jobs.X.attr'
key, so a key like 'jobs.fakefoo.max_strikes' passed validation, was
persisted to config_overrides, then silently no-op'd at apply time
because settings.jobs has no 'fakefoo'. The bogus row stayed in the DB
and was re-read on every restart, polluting the override set forever.

Validate that parts[1] resolves to an actual job object on settings.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously the migration ran via executescript(), and the
schema_version row was only updated after the whole loop finished.
If a future migration failed halfway through, partial DDL would be
applied but schema_version would still point at the old version,
making the next startup either re-apply already-applied statements
(fine for CREATE TABLE IF NOT EXISTS, broken for ALTER TABLE) or
silently skip the rest.

Seed the schema_version row up front, then append the version bump
to each migration's script so the version is updated as part of the
same executescript call. SQLite commits each script implicitly, so
either the migration applies and version moves, or neither does.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The SQL fetch for UI-protected download IDs was inside the for-loop
over arr instances, so it ran N times per request even though the
result is identical for every iteration. Hoist it to a single query
before the loop.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
uvicorn was hardcoded to log_level=debug regardless of the app's
configured log level, flooding production logs with framework noise.
Translate settings.general.log_level into the uvicorn equivalent and
default to 'info' for unknowns (including the app's custom VERBOSE
level, which uvicorn doesn't recognize).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
asyncio.create_task() returns a Task that the event loop only holds a
weak reference to (per the docs). Discarding the return value lets the
GC collect the task, which can cause it to disappear mid-flight. Both
_mark_first_cycle_done and _periodic_cleanup were fire-and-forget here.

Store the Task handles on app.state so they live as long as the app.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
api_unprotect deleted the protected_downloads row but never emitted an
event, so other browser tabs subscribed to the SSE stream stayed out
of date and the activity log silently dropped unprotect actions
(only protect was recorded).

Add an ITEM_UNPROTECTED EventType, emit it from api_unprotect with
the title/arr_name we look up before the delete, record it in
ActivityRecorder's action_map as 'unprotected', and add the option
to the activity page filter dropdown.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The web UI exposes runtime mutation endpoints (toggle test_run,
disable jobs, change min_speed, trigger cycles, delete protected rows)
with no built-in authentication. The PR thread treats this as the
user's responsibility, but the README didn't surface that.

Add a Security subsection recommending bind to localhost + SSH tunnel,
reverse proxy auth, or disabling the UI; warn against exposing
port 9999 directly to the public internet.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Embedding the listen port in the proxy path matched code-server's
/proxy/<port>/ convention but produced wrong root_path values for users
behind nginx/Traefik/Caddy, who expect a clean prefix like /decluttarr
without the port appended.

Build root_path as f"/{proxy_prefix.strip('/')}" instead. Code-server
users now write the port into the prefix value itself
(proxy_prefix: "proxy/9999"); other proxy users get the clean prefix
they want. Strip slashes defensively so leading/trailing variations all
behave the same.

This is a breaking change for anyone who had configured proxy_prefix
under the old format — README and config_example updated to explain
the new behavior. ManiMatter approved option 2 in the PR thread.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the inline style attribute on the activity pager into a reusable
.pagination class with explicit gap and padding-block. Per Dark3clipse's
review screenshot the prior layout felt cramped — adding gap and giving
the page-counter span flex: 1 1 auto with centered text gives the
buttons proper breathing room and keeps the counter centered as the
container width changes.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per Dark3clipse's review the dashboard felt cramped between the
instance cards, the action controls, and the queue/activity sections.
Add a .dashboard-actions class (replacing the inline style on the
Actions row) with margin-block: 1.5rem, and apply the same vertical
margin to <section> elements so the Download Queue and Recent Activity
blocks each get breathing room.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously the SQLite path was only configurable via the
DECLUTTARR_DB_PATH environment variable. Surface it in the YAML
config too as web.db_path so users running with a config file (the
recommended path) can point it at a dedicated volume or PVC without
also setting an env var.

Precedence: web.db_path (YAML) > DECLUTTARR_DB_PATH (env) > default
"./data/decluttarr.db". Database() already accepted an optional
db_path arg, so this is just plumbing the new setting through and
adding documentation. Addresses Dark3clipse's review point on
configurable database location.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the empty data: URI in base.html with a real SVG favicon.
Going with a broom emoji thematic to the project's purpose (cleaning
download queues). It's a single self-contained .svg file in static/,
no additional dependencies, and easy to swap later if a custom design
is preferred.

ManiMatter approved the addition in the PR thread; addresses
Dark3clipse's review point on browser-tab branding.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…y load

_load_from_env stored env-var keys lowercased verbatim, so WEB_HOST
became {"web": {"web_host": ...}} while Web.__init__ reads
web_config.get("host", ...). Result: WEB_ENABLED, WEB_HOST, WEB_PORT,
WEB_DB_PATH all silently no-op'd when configured via environment
variables — the values were loaded into the dict but under keys
nothing actually reads. PROXY_PREFIX was the only web env var that
worked, because its name doesn't start with the WEB_ prefix.

Practical impact: env-var deployments could not change the listen
address, port, database path, or even disable the web UI — every
WEB_ENABLED=false ran the UI anyway. Verified empirically before fix.

Strip the section-name prefix from env-var keys when storing in
_load_from_env so they match the YAML schema. This is surgical:
no other section uses an env-var prefix matching its name, so the
behavior of general/jobs/instances/download_clients is unchanged.

Add a parametrized regression test covering all five web env vars,
including a value (WEB_ENABLED=false) that differs from the default
so a no-op fix would be caught.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace CDN-hosted Pico CSS / HTMX / Alpine.js with local copies under
src/web/static/vendor/. Removes runtime dependency on jsdelivr.com and
unpkg.com, which means:

  - works in airgapped / restricted-egress environments
  - no need to allow CDN hosts in CSP script-src / style-src
  - no DNS round-trip on first paint
  - SRI hashes no longer needed (version pinning + local serving
    provides the same integrity guarantee)

Versions pinned:
  - Pico CSS 2.1.1 (latest 2.x at vendor time)
  - HTMX 2.0.4 (matched existing CDN pin)
  - Alpine.js 3.14.8 (matched existing CDN pin)

The downloaded files' sha-384 hashes match the existing integrity=
attributes byte-for-byte, so this is a pure host-swap with zero
behavior change. Addresses Dark3clipse's review point on external CDN
dependencies.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the root-path / htmx-prefix wiring out of an inline <script> block
in base.html into static/base.js. The Jinja-rendered root_path value
moves into a <meta name="root-path"> tag, which is a CSP-clean HTML
attribute (no script execution). base.js reads it via querySelector.

Net effect for CSP: this template no longer needs 'unsafe-inline' in
script-src.

base.js loads synchronously before htmx.min.js so its
htmx:configRequest listener is registered before htmx fires the event.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the protect/unprotect click handler and the dashboard() Alpine
factory (SSE wiring + Run Now / Toggle Test Run actions) out of the
inline <script> block in dashboard.html into static/dashboard.js. The
template now just <script src=...>'s the file.

No behavior change. Removes another inline-script source so script-src
CSP can drop 'unsafe-inline' for this page.

Load order: dashboard.js loads synchronously at the bottom of <body>
(via the {% block scripts %} slot in base.html), which runs during
parsing — before Alpine's deferred init walks the DOM looking for
x-data="dashboard()", so the factory is defined when Alpine needs it.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two CSP-relevant changes here:

1. Move the settingsPage() Alpine factory + flash animation logic out
   of the inline <script> block in settings.html into static/settings.js.

2. Replace the two <script type="application/json"> blocks (which CSP
   blocks under script-src even though they aren't executable) with
   data-config / data-overrides attributes on a hidden
   #settings-init-data div. Same Jinja autoescape, just an HTML
   attribute instead of a script tag, so script-src doesn't apply.

settings.js reads the data attrs at factory-call time and parses with
JSON.parse, returning the populated state object so Alpine's initial
binding evaluation has the data it needs (avoids null-deref on
x-model="config.general.test_run" before init).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move the activityLog() Alpine factory (filter state + paginated
/api/activity fetch + timestamp formatting) out of the inline <script>
block into static/activity.js. Same load-order story as the other
extractions: external script in {% block scripts %} executes during
body parse, before Alpine's deferred init.

This is the last template-level inline <script>. Combined with the
previous extractions, the only remaining script tags in the rendered
HTML are <script src="..."> references — no inline content.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a CSP subsection to the web UI README documenting the policy the
UI works under after the Tier 2 changes (vendored deps, no inline
scripts, no inline JSON data blocks):

  default-src 'self';
  script-src  'self' 'unsafe-eval';
  style-src   'self';
  connect-src 'self';
  img-src     'self' data:;

Two notable properties:

- No external CDN allowlist needed — Pico/HTMX/Alpine are all vendored.
- 'unsafe-inline' is not required for script-src — all JS is in
  external .js files now.

The only relaxation we still need is 'unsafe-eval' because Alpine.js
compiles x-data / x-show / x-text expressions with new Function(...).
A follow-up PR can migrate to the alpinejs-csp build to drop that
requirement; this README note flags the dependency explicitly so users
deploying behind strict CSP know what to allow.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@ManiMatter

Copy link
Copy Markdown
Owner

@Dark3clipse - just checking if you are still planning on doing a review on this PR? would love this to be merged but unfortunately don't have the time to review it myself

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants