Merge branch 'pr/NaruZosa/252' into decluttarr-v2

Author: ManiMatter

.gitignore

@@ -9,4 +9,5 @@ venv
 temp
 .notebooks
 **/old/
-logs/*
+logs/*
+.idea/*

.vscode/settings.json

@@ -6,5 +6,5 @@
         "tests"
     ],
     "python.testing.unittestEnabled": false,
-    "python.testing.pytestEnabled": true,
+    "python.testing.pytestEnabled": true
 }

CONTRIBUTING.md

@@ -2,9 +2,9 @@
 
 ## Table of contents
 - [Overview](#overview)
-- [Feature Requests](#feature--requests)
-- [Bug Reports](#bug--reports)
-- [Code Contributions](#code--contributions)
+- [Feature Requests](#feature-requests)
+- [Bug Reports](#bug-reports)
+- [Code Contributions](#code-contributions)
 
 ## Overview
 Thank you for wanting to contribute to this project.
@@ -27,7 +27,7 @@ Please go follow these steps to submit a bug:
 - Create meaningful logs by:
 1) Switch decluttarr to debug mode (setting LOG_LEVEL: DEBUG)
 2) Turn off all remove functions but one where you expect a removal (example: REMOVE_STALLED: True and the rest on False)
-3) Let it run until the supposed remove should be trigged
+3) Let it run until the supposed remove should be triggered
 4) Paste the full logs to a pastebin
 5) Share your settings (docker-compose or config.conf) 
 6) Optional: If helpful, share screenshots showing the problem (from your arr-app or qbit) 
@@ -40,23 +40,23 @@ Code contributions are very welcome - thanks for helping improve this app!
 3) Only commit code that you have written yourself and is not owned by anybody else
 4) Create a PR against the "dev" branch
 5) Be responsive to code review 
-5) Once the code is reviewed and OK, it will be merged to dev branch, which will create the "dev"-docker image
-6) Help testing that the dev image works
-7= Finally, we will then commit the change to the main branch, which will create the "latest"-docker image
+6) Once the code is reviewed and OK, it will be merged to dev branch, which will create the "dev"-docker image
+7) Help testing that the dev image works
+8) Finally, we will then commit the change to the main branch, which will create the "latest"-docker image
 
 You do not need to know about how to create docker images to contribute here.
 To get started:
 1) Create a fork of decluttarr
 2) Clone the git repository from the dev branch to your local machine `git clone -b dev https://github.com/yourName/decluttarr`
-2) Create a virtual python environment (`python3 -m venv venv`)
-3) Activate the virtual environment (`source venv/bin/activate`)
-4) Install python libraries (`pip install -r docker/requirements.txt`)
-5) Adjust the config/config.conf to your needs
-6) Adjust the code in the files as needed
-7) Run the script (`python3 main.py`)
-8) Push your changes to your own git repo and use a descriptive name for the branch name (e.g. add-feature-to-xyz; bugfix-xyz)
-9) Test the dev-image it creates automatically
-10) Create the PR from your repo to ManiMatter/decluttarr (dev branch)
-11) Make sure all checks pass
-12) Squash your commits
-13) Test that the docker image works that was created when you pushed to your fork 
+3) Create a virtual python environment (`python3 -m venv venv`)
+4) Activate the virtual environment (`source venv/bin/activate`)
+5) Install python libraries (`pip install -r docker/requirements.txt`)
+6) Adjust the config/config.conf to your needs
+7) Adjust the code in the files as needed
+8) Run the script (`python3 main.py`)
+9) Push your changes to your own git repo and use a descriptive name for the branch name (e.g. add-feature-to-xyz; bugfix-xyz)
+10) Test the dev-image it creates automatically
+11) Create the PR from your repo to ManiMatter/decluttarr (dev branch)
+12) Make sure all checks pass
+13) Squash your commits
+14) Test that the docker image works that was created when you pushed to your fork 

README.md

@@ -11,7 +11,7 @@ _Like this app? Thanks for giving it a_ ⭐️
     - [Docker-compose with config file (recommended)](#docker-docker-compose-together-with-configyaml)
     - [Docker-compose only](#docker-specifying-all-settings-in-docker-compose)
 - [Explanation of the settings](#explanation-of-the-settings)
-  - [General](#general)
+  - [General](#general-settings)
     - [LOG_LEVEL](#log_level)
     - [TEST_RUN](#test_run)
     - [TIMER](#timer)
@@ -36,7 +36,7 @@ _Like this app? Thanks for giving it a_ ⭐️
     - [REMOVE_UNMONITORED](#remove_unmonitored)
     - [SEARCH_CUTOFF_UNMET_CONTENT](#search_unmet_cutoff_content)
     - [SEARCH_MISSING_CONTENT](#search_missing_content)
-  - [Instances](#instances)
+  - [Instances](#arr-instances)
     - [SONARR](#sonarr)
     - [RADARR](#radarr)
     - [READARR](#readarr)
@@ -64,7 +64,7 @@ Feature overview:
 -   Removing downloads that are stalled (remove_stalled)
 -   Removing downloads belonging to movies/series/albums etc. that have been marked as "unmonitored" (remove_unmonitored)
 -   Periodically searching for better content on movies/series/albums etc. where cutoff has not been reached yet (search_cutoff_unmet_content)
--   Periodcially searching for missing content that has not yet been found (search_missing_content)
+-   Periodically searching for missing content that has not yet been found (search_missing_content)
 
 
 Key behaviors:
@@ -91,7 +91,7 @@ How to run this:
     -   The feature that distinguishes private and private trackers (private_tracker_handling, public_tracker_handling) does not work
     -   Removal of bad files and <100% availability (remove_bad_files) does not work 
 -   If you see strange errors such as "found 10 / 3 times", consider turning on the setting "Reject Blocklisted Torrent Hashes While Grabbing". On nightly Radarr/Sonarr/Readarr/Lidarr/Whisparr, the option is located under settings/indexers in the advanced options of each indexer, on Prowlarr it is under settings/apps and then the advanced settings of the respective app
--   If you use qBittorrent and none of your torrents get removed and the verbose logs tell that all torrents are protected by the protected_tag even if they are not, you may be using a qBittorrent version that has problems with API calls and you may want to consider switching to a different qBit image (see https://github.com/ManiMatter/decluttarr/issues/56)
+-   If you use qBittorrent and none of your torrents get removed and the verbose logs tell that all torrents are protected by the protected_tag even if they are not, you may be using a qBittorrent version that has problems with API calls, and you may want to consider switching to a different qBit image (see https://github.com/ManiMatter/decluttarr/issues/56)
 -   Currently, “\*Arr” apps are only supported in English. Refer to issue https://github.com/ManiMatter/decluttarr/issues/132 for more details
 -   If you experience yaml issues, please check the closed issues. There are different notations, and it may very well be that the issue you found has already been solved in one of the issues. Once you figured your problem, feel free to post your yaml to help others here: https://github.com/ManiMatter/decluttarr/issues/173
 
@@ -125,13 +125,13 @@ jobs:
 ### Running in docker
 
 In docker, there are two ways how you can run decluttarr.
-The [recommendeded approach](#docker-docker-compose-together-with-configyaml) is to use a config.yaml file (similar to running the script [locally](#running-locally)).
+The [recommended approach](#docker-docker-compose-together-with-configyaml) is to use a config.yaml file (similar to running the script [locally](#running-locally)).
 Alternatively, you can put all settings [directly in your docker-compose](#docker-specifying-all-settings-in-docker-compose), which may bloat it a bit.
 
 
 #### Docker: Docker-compose together with Config.yaml
 1. Use the following input for your `docker-compose.yml`
-2. Download the config_example.yaml from the config folder (on github) and put it into your mounted folder
+2. Download the config_example.yaml from the config folder (on GitHub) and put it into your mounted folder
 3. Rename it to config.yaml and adjust the settings to your needs
 4. Run `docker-compose up -d` in the directory where the file is located to create the docker container
 
@@ -162,7 +162,7 @@ If you want to have everything in docker compose:
 1. Use the following input for your `docker-compose.yml`
 2. Tweak the settings to your needs
 3. Remove the things that are commented out (if you don't need them), or uncomment them
-4. If you face problems with yaml formats etc, please first check the open and closed issues on github, before opening new ones
+4. If you face problems with yaml formats etc., please first check the open and closed issues on GitHub, before opening new ones
 5. Run `docker-compose up -d` in the directory where the file is located to create the docker container
 
 Note: Always pull the "**latest**" version. The "dev" version is for testing only, and should only be pulled when contributing code or supporting with bug fixes
@@ -207,7 +207,7 @@ services:
       SEARCH_MISSING_CONTENT: True
 
       # # --- OR: Jobs (with job-specific settings) ---
-      # Alternatively, you can use the below notation, which for certain jobs allows you to set additioanl parameters
+      # Alternatively, you can use the below notation, which for certain jobs allows you to set additional parameters
       # As written above, these can also be set as Job Defaults so you don't have to specify them as granular as below.
       # REMOVE_BAD_FILES: |
       #   keep_archives: True
@@ -314,14 +314,14 @@ Configures the general behavior of the application (across all features)
 -   Allows you to configure download client names that will be skipped by decluttarr
     Note: The names provided here have to 100% match with how you have named your download clients in your *arr application(s)
 -   Type: List of strings
--   Is Mandatory: No (Defaults to [], ie. nothing ignored])
+-   Is Mandatory: No (Defaults to [], i.e. nothing ignored])
 
 #### PRIVATE_TRACKER_HANDLING / PUBLIC_TRACKER_HANDLING
 
 -   Defines what happens with private/public tracker torrents if they are flagged by a removal job
 -   Note that this only works for qbittorrent currently (if you set up qbittorrent in your config)
     -   "remove" means that torrents are removed (default behavior)
-    -   "skip" means they are disregarded (which some users might find handy to protect their private trackers prematurely, ie., before their seed targets are met)
+    -   "skip" means they are disregarded (which some users might find handy to protect their private trackers prematurely, i.e., before their seed targets are met)
     -   "obsolete_tag" means that rather than being removed, the torrents are tagged. This allows other applications (such as [qbit_manage](https://github.com/StuffAnThings/qbit_manage) to monitor them and remove them once seed targets are fulfilled)
 -   Type: String
 -   Permissible Values: remove, skip, obsolete_tag
@@ -372,10 +372,10 @@ If a job has the same settings configured on job-level, the job-level settings w
 #### MAX_CONCURRENT_SEARCHES
 
 -   Only relevant together with search_unmet_cutoff_content and search_missing_content
--   Specified how many ites concurrently on a single arr should be search for in a given iteration
+-   Specified how many ites concurrently on a single arr should be searched for in a given iteration
 -   Each arr counts separately
 -   Example: If your wanted-list has 100 entries, and you define "3" as your number, after roughly 30 searches you'll have all items on your list searched for.
--   Since the timer-setting steer how often the jobs run, if you put 10minutes there, after one hour you'll have run 6x, and thus already processed 18 searches. Long story short: No need to put a very high number here (else you'll just create unecessary traffic on your end..).
+-   Since the timer-setting steer how often the jobs run, if you put 10minutes there, after one hour you'll have run 6x, and thus already processed 18 searches. Long story short: No need to put a very high number here (else you'll just create unnecessary traffic on your end.).
 -   Type: Integer
 -   Permissible Values: Any number
 -   Is Mandatory: No (Defaults to 3)
@@ -389,12 +389,12 @@ This is the interesting section. It defines which job you want decluttarr to run
 - Steers whether files within torrents are marked as 'not download' if they match one of these conditions
   1) They are less than 100% available
   2) They are not one of the desired file types supported by the *arr apps:
-  3) They contain one of these words (case insensitive) and are smaller than 500 MB:
+  3) They contain one of these words (case-insensitive) and are smaller than 500 MB:
      - Trailer
      - Sample
 
 -   If all files of a torrent are marked as 'not download' then the torrent will be removed and blacklisted
--   Note that this is only supported when qBittorrent is configured in decluttarr and it will turn on the setting 'Keep unselected files in ".unwanted" folder' in qBittorrent 
+-   Note that this is only supported when qBittorrent is configured in decluttarr, and it will turn on the setting 'Keep unselected files in ".unwanted" folder' in qBittorrent 
 -   Type: Boolean or Dict
 -   Permissible Values: True, False or keep_archives (bool)
 -   Is Mandatory: No (Defaults to False)
@@ -431,8 +431,8 @@ This is the interesting section. It defines which job you want decluttarr to run
 -   Permissible Values: True, False or max_strikes (int)
 -   Is Mandatory: No (Defaults to False)
 -   Note:
-      - With max_strikes you can define how many time this torrent can be caught before being removed
-      - Instead of configuring it here, you may also configure it as a default across all jobs or use the built-in defaults (see futher above under "Max_Strikes")
+      - With max_strikes you can define how many times this torrent can be caught before being removed
+      - Instead of configuring it here, you may also configure it as a default across all jobs or use the built-in defaults (see further above under "Max_Strikes")
 
 #### REMOVE_MISSING_FILES
 
@@ -455,7 +455,7 @@ This is the interesting section. It defines which job you want decluttarr to run
 
 -   Steers whether slow downloads are removed from the queue
 -   Blocklisted: Yes
--   Note: Does not apply to usenet downloads (since there users pay for certain speed, slowness should not occurr)
+-   Note: Does not apply to usenet downloads (since there users pay for certain speed, slowness should not occur)
 -   Type: Boolean or Dict
 -   Permissible Values: 
       If bool: True, False 

config/config_example.yaml

@@ -58,7 +58,7 @@ instances:
 
 download_clients:
   qbittorrent:
-    - base_url: "http://qbittorrent:8080" # You can use decluttar without qbit (not all features available, see readme).
+    - base_url: "http://qbittorrent:8080" # You can use decluttarr without qbit (not all features available, see readme).
       # username: xxxx # (optional -> if not provided, assuming not needed)
       # password: xxxx # (optional -> if not provided, assuming not needed)
       # name: "qBittorrent" # (optional -> if not provided, assuming "qBittorrent". Must correspond with what is specified in your *arr as download client name)

docker/dockerfile

@@ -37,7 +37,7 @@ COPY main.py main.py
 COPY src src
 
 
-# Install health check
+# Install health check
 RUN apt-get update && apt-get install -y --no-install-recommends procps && \
     apt-get clean && rm -rf /var/lib/apt/lists/*
 HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \

docker/requirements.txt

@@ -11,3 +11,4 @@ autoflake==2.3.1
 isort==5.13.2
 envyaml==1.10.211231
 demjson3==3.0.6
+ruff==0.11.11

main.py

@@ -1,9 +1,9 @@
 import asyncio
-from src.settings.settings import Settings
 
-from src.utils.startup import launch_steps
-from src.utils.log_setup import logger
 from src.job_manager import JobManager
+from src.settings.settings import Settings
+from src.utils.log_setup import logger
+from src.utils.startup import launch_steps
 
 settings = Settings()
 job_manager = JobManager(settings)

pyproject.toml

@@ -1,5 +1,5 @@
 [tool.pylint]
-ignore = ".venv"
+ignore = [".venv", "old"]
 ignore-patterns = ["__pycache__", ".pytest_cache"]
 disable = [
     "logging-fstring-interpolation",    # W1203
@@ -9,6 +9,7 @@ disable = [
     "missing-class-docstring",          # C0115
     "missing-function-docstring",       # C0116
     "line-too-long",                    # C0301
+    "too-few-public-methods",           # R0903
 ]
 
 [tool.pytest.ini_options]

ruff.toml

@@ -0,0 +1,54 @@
+# Exclude a variety of commonly ignored directories.
+exclude = [
+    ".bzr",
+    ".direnv",
+    ".eggs",
+    ".git",
+    ".git-rewrite",
+    ".hg",
+    ".ipynb_checkpoints",
+    ".mypy_cache",
+    ".nox",
+    ".pants.d",
+    ".pyenv",
+    ".pytest_cache",
+    ".pytype",
+    ".ruff_cache",
+    ".svn",
+    ".tox",
+    ".venv",
+    ".vscode",
+    "__pypackages__",
+    "_build",
+    "buck-out",
+    "build",
+    "dist",
+    "node_modules",
+    "site-packages",
+    "venv",
+]
+
+# Assume Python 3.10
+target-version = "py310"
+
+[lint]
+# Enable Pyflakes (`F`) and a subset of the pycodestyle (`E`) codes by default.
+# Unlike Flake8, Ruff doesn't enable pycodestyle warnings (`W`) or
+# McCabe complexity (`C901`) by default.
+select = ["ALL"]
+ignore = ["D203", "D212", "E501"]
+
+[lint.per-file-ignores]
+# "src/jobs/remove_bad_files.py" = ["ERA001"]
+"tests/settings/test__user_config_from_env.py" = ["S101"]
+"tests/jobs/test_strikes_handler.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_unmonitored.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_stalled.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_slow.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_orphans.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_missing_files.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_metadata_missing.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_failed_imports.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_failed_downloads.py" = ["S101", "SLF001"]
+"tests/jobs/test_remove_bad_files.py" = ["S101", "SLF001"]
+"tests/jobs/test_removal_handler.py" = ["S101", "SLF001"]