[pull] main from mhdzumair:main#267
Merged
Merged
Conversation
…nd advanced import
- Implement list_downloaded_torrents for all debrid providers (Premiumize,
OffCloud, PikPak, Seedr) alongside existing RD/TorBox/AllDebrid/Debrid-Link
- Parse torrent titles with PTT; detect movie/series/sports and match against
the media DB to populate matched_title + external_ids (imdb/tmdb/tvdb)
- Add dedicated Formula/MotoGP racing parser: series title "{league} {event}
{year}" with the session (Qualifying/Race/Sprint) as the episode
- Surface detected sports_category in the missing-torrents response so the UI
can preselect the category for quick import
- Rewrite advanced import to match the frontend payload (advanced_imports);
create/resolve media (sports stubs as series for F1/MotoGP, movie otherwise),
link streams, per-file episode metadata, languages, and catalogs
…for racing Frontend: - Add sports_category to MissingTorrentItem so the detected category preselects in the Edit Metadata panel and Advanced Import dialog - Show the file/episode annotation button for racing-series sports (formula_racing / motogp_racing), not just movie/series - Carry episode_title / release_date through the advanced-import file payload Backend: - Accept episode_title / release_date on advanced-import file entries - Populate series_metadata / season / episode rows during advanced import so series (incl. F1/MotoGP) list their episodes on the detail page, using the annotated episode names
…ts response - Derive the racing session name (e.g. "Qualifying") per video file from its filename and include it as files[].episode_title for F1/MotoGP torrents - Frontend: thread episode_title through MissingTorrentFile → TorrentFile so the annotation dialog prefills the episode title for racing series - Add test for dotted filename with extension
…tine Fixes a NOT-NULL violation (is_remastered/is_upscaled/is_subbed) in the debrid advanced import, which used an incomplete stream insert. - Add import_helpers::persist_torrent_import — the single routine that writes a torrent stream + torrent_stream row (all columns), links media, trackers, languages, audio/HDR/channel extras, per-file metadata (with season/episode links), catalogs, and series season/episode metadata - Add import_helpers::insert_torrent_stream_row and ensure_series_episode_metadata - Route watchlist advanced import, content magnet import, and content torrent file import through the shared routine; remove the duplicated per-path inserts - Content imports now also populate series season/episode metadata As a result, series (incl. F1/MotoGP) consistently list episodes on the detail page regardless of import route.
…mport analysis
Episode organization (applies to any non-IMDb/user-created series during import):
- Racing (F1/MotoGP) sessions map to canonical fixed slots: FP1, FP2/Sprint
Qualifying, FP3/Sprint, Qualifying, Race — stable across re-imports
- Files with a detectable date are ordered chronologically; dates already present
reuse their episode number to align with existing media
- Remaining files are numbered after the current max, by filename order
- Files with explicit episode numbers and provider-backed series are left untouched
- Runs inside persist_torrent_import, so every import route benefits
Quick import (POST /watchlist/{provider}/import) rewritten:
- Accept the real payload (info_hashes + overrides + anon fields) instead of the
stale {items} shape
- Re-fetch torrent files from the provider, match movies/series against the
metadata DB for an external id, create stubs for sports, then persist via the
shared routine (with episode organization)
Add parser::racing_session_episode and import_helpers::{extract_iso_date,
organize_user_series_episodes}; remove the now-unused upsert_debrid_torrent.
Two silent failures meant imported sports/user-created series showed no episodes and no catalog: - series_metadata/season/episode IDs are `integer` (i32), but the inserts decoded RETURNING id as i64 — a runtime decode error that was swallowed, so series_id came back None and episode creation bailed out. Fixed the column types. - catalog has NOT NULL `is_system`/`display_order` with no default; the catalog upsert only set `name`, so it errored (silently). Now sets is_system=false, display_order=0. Also surface these failures via tracing::warn instead of swallowing them.
Implements the full Telegram contribution bot in src/bot/ (18 modules), fixes the broken MTProto session loader, and wires per-user channels. Bot (src/bot/): - Webhook dispatcher routes messages and callbacks to typed handlers - /start /help /login /status /cancel /scrape commands - Multi-step wizard: detect content type → media type → analyze → match selection (inline keyboard) → metadata review (edit resolution/quality/codec) → confirm import - All content types: magnet, torrent file/URL, NZB, YouTube, HTTP, AceStream, forwarded video, sports - Anonymous display name prompt for contribute_anonymously users - Batch mode: forward multiple files, per-item review buttons, "Set as TV Series" auto-assigns series/season/episode from filenames, Import All for bulk ingestion - Login token generation (consumed by existing telegram_login endpoint) - Redis-backed conversation/batch state with 30-min / 24-h TTLs - Callback payload caching for 64-byte Telegram limit - Backup channel copy via TELEGRAM_BACKUP_CHANNEL_ID Wiring: - telegram_webhook.rs: parses Update, spawns dispatch, fast-acks - register_commands (setMyCommands) on API startup - All imports go through process_contribution_import pipeline (contribution record, auto-approve, points, moderator notify) Session fix (src/scrapers/telegram.rs + src/util/telegram_session.rs): - Replaces non-functional bincode_deserialize stub with real Telethon StringSession parser (dc_id, ip, port, auth_key extraction) - init_client fails loudly when session is unauthenticated instead of silently scraping nothing - New binary src/bin/telegram_session.rs for session conversion Per-user channels (src/db/telegram_channels.rs + orchestrator.rs): - Reads enabled channels from profile tgc config - Request-time and background scrapers both pass per-user channels Import handler refactors: - torrent_import: pub analyze_magnet_for_bot, analyze_torrent_bytes, extract_info_hash_from_magnet - youtube_import: pub analyze_youtube_for_bot - nzb_import: pub analyze_nzb_url_for_bot - http_import: pub analyze_http_for_bot - acestream_import: pub analyze_acestream_for_bot Config: adds telegram_backup_channel_id (TELEGRAM_BACKUP_CHANNEL_ID)
…acker Tier 1 — SQL type/enum hard errors (broken endpoints): - watch_history.rs: cast action param as ::watchaction in dynamic WHERE - user_library.rs: decode RETURNING id/media_id as i32 (INT4 not INT8) - rss.rs: decode rss_feed.id as i32 in user_update_rss_feed check - content/streams.rs: SELECT stream_type::text to decode streamtype enum - content/user_metadata.rs: fix ::media_type_enum → ::mediatype; fix series_id/season_id/genre_id/media_id/creator_id to i32; add missing mediatype cast in list query too - admin_scrapers.rs: remove invalid ON CONFLICT on UPDATE; guard with NOT EXISTS subquery; bind ids as i32 Tier 2 — Data integrity bugs & panic: - db/streams.rs: supply required NOT-NULL cols in file_media_link INSERT (is_primary, confidence, link_source, created_at) - ptt/engine.rs: use floor_char_boundary for all byte-index string slices to prevent scraper task panics on multibyte UTF-8 titles - scrapers/persist.rs: strip NUL bytes (0x00) from usenet stream text fields before Postgres insert; add strip_nul() helper Tier 3 — HTTP 414 / oversized GET requests: - premiumize.rs: batch check_cached into chunks of 80 hashes per request - stremthru.rs: batch check_cached into chunks of 80 hashes per request Tier 4 — Log noise reduction: - providers/mod.rs: treat reqwest decode errors as transient (→ debug), collapsing 138k playback.rs warnings - torrents/torbox.rs: check HTTP status before JSON decode; downgrade checkcached decode failure to debug, collapsing 45k warnings - routes/poster.rs: downgrade upstream 404 and annotate-failed to debug - jobs/runner.rs: use PgListener::connect() (dedicated standalone conn) instead of connect_with(&pool) to prevent NotificationResponse on shared pool query connections
Instead of silently propagating reqwest's opaque "error decoding response body" error, add a shared response_json() helper in providers/mod.rs that reads the response as text first, then parses JSON, and warns with the HTTP status + truncated body preview (up to 500 chars) on failure. This replaces the previous approach of treating decode errors as transient which hid the problem. Now every decode failure produces a WARN with the exact response from the debrid service (rate-limit page, HTML 502, plain-text error, etc.) so the root cause is visible in the exception tracker. Also reverts the providers/mod.rs is_decode() change. Applied to all provider HTTP helpers: rd_get/rd_post/get_access_token (realdebrid), tb_get/tb_post_form/tb_post_json/checkcached (torbox), ad_get/ad_post_form/magnet-status (alldebrid), pm_post_form (premiumize), st_get/st_post/magnets-check (stremthru), oc_get/oc_post_form/cache (offcloud), dl_get/dl_post/refresh-token/seedbox-list (debridlink), link-generate/link-lookup (easydebrid).
Root cause of recurring i64/INT4 and enum-cast runtime errors:
- ~1,124 runtime (unchecked) sqlx queries vs 13 compile-time macros
- 240 i64 fields where schema has zero bigint id/FK columns (all INT4)
- 14 Postgres enums handled as String + 112/216 hand-written SQL casts
- CI had no Postgres, so type drift was never caught at build time
This commit establishes the foundation that makes recurrence structurally
harder and detectable at compile/CI time:
src/db/types.rs (new)
- All 14 Postgres enum types as native Rust enums with #[derive(sqlx::Type)]
and #[sqlx(type_name=...)] — bind/decode without any ::enumname or ::text
cast; wrong variant is a compile error
- 10 core ID newtypes (MediaId, StreamId, UserId, ProfileId, StreamFileId,
SeriesId, SeasonId, EpisodeId, GenreId, IntegrationId) with #[sqlx(transparent)]
forcing i32; documents which 8 columns are legitimately i64
- Re-exported from src/db/mod.rs for codebase-wide use
- Cargo.toml: add explicit "macros" feature to sqlx
.github/workflows/ci.yml
- Add postgres:17-alpine service to rust-build job
- Install sqlx-cli; run migrations; run `cargo sqlx prepare --check`
→ CI now fails if any query! macro drifts from the real schema
- Build and test with SQLX_OFFLINE=true + TEST_DATABASE_URL wired in
.githooks/pre-commit (new, version-controlled)
- Adds cargo check (SQLX_OFFLINE=true) for staged Rust files, alongside
existing ruff/prettier/eslint checks
- Session already updated via `make install-hooks`
Makefile: add `install-hooks` target
AGENTS.md: fix false "all queries compile-time validated" claim;
document INT4/i64 rules, enum type rules, and developer workflow
Next step: migrate static queries in src/db/ layer to query_as! macros
using the new types (requires `cargo sqlx prepare` against a live DB).
issues
Main changes (Cursor migration + manual fixes):
- Replace all string-cast WHERE comparisons (type::text = 'MOVIE',
status::text = $N,
format!("m.type::text = '{...}'")) with native enum binds — no cast,
no injection risk
- Convert db/meta.rs Internal branch to query_as! macro (compile-time
SQL verification,
mirrors External branch; collapses duplicated base_sql)
- Port contribution bot handlers (bot/) to Rust with full parity to
Python
- Update all DB query result types to use MediaId/UserId/MediaType
newtypes so wrong
types cause compile errors rather than silent runtime failures
- Add 13 new .sqlx/ entries for offline SQLX compile-time query
verification
- Fix configure.rs: preserve secret_str through redirect to
/app/configure?secret_str=
- Fix metrics_middleware.rs: RAII InFlightGuard prevents stuck in-flight
counter on
client disconnect / future cancellation
- Fix easynews.rs: 401 short-circuits all remaining queries
(user-specific cred failure)
- Fix torbox_search.rs: 429 skips title query and returns early
(per-token rate limit)
- Fix public_indexer_registry.rs: sort non-CF indexers first so byparr
sites run last
- Disable CF bypass (byparr) in live API fan_out — Chromium sessions
only in bg workers
- Add source health gate (NZBIndex + Binsearch) — 429s record failures;
gate blocks
further calls once success rate falls below threshold
- Fix /metrics column bug (media_type → type); replace COUNT(*) full
table scans on
large stream tables with pg_stat_user_tables n_live_tup estimates
- Remove unauthenticated /metrics root route; keep /api/v1/metrics under
api_key_middleware
- Log Zilean response body on JSON parse failure (was silently dropping
error context)
- Add Grafana + Prometheus dev stack (docker-compose-grafana-dev.yml,
prometheus-dev.yml)
Switch checkcached to the POST batch endpoint for larger hash lists and return requestdl URLs with redirect=true so playback skips a server-side JSON round trip.
Load stored torrent bytes from the DB and submit them to TorBox when available, falling back to magnet links for streams without a stored file.
…arity Store torrent_type and torrent_file from Prowlarr/Jackett/RSS scrapes, link announce trackers, gate catalog streams by provider, and prefer file upload on debrid playback. Fix feed-job privacy bypass, download URL fallback, flag parsing, upload corruption, and bounded torrent fetches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )