feat: add configurable timeouts (#673)

## ℹ️ Description
- Related issues: #671, #658
- Introduces configurable timeout controls plus retry/backoff handling
for flaky DOM operations.

We often see timeouts which are note reproducible in certain
configurations. I suspect timeout issues based on a combination of
internet speed, browser, os, age of the computer and the weather.

This PR introduces a comprehensive config model to tweak timeouts.

## 📋 Changes Summary
- add TimeoutConfig to the main config/schema and expose timeouts in
README/docs
- wire WebScrapingMixin, extractor, update checker, and browser
diagnostics to honor the configurable timeouts and retries
- update translations/tests to cover the new behaviour and ensure
lint/mypy/pyright pipelines remain green

### ⚙️ Type of Change
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x] ✨ New feature (adds new functionality without breaking existing
usage)
- [ ] 💥 Breaking change (changes that might break existing user setups,
scripts, or configurations)

## ✅ Checklist
- [x] I have reviewed my changes to ensure they meet the project's
standards.
- [x] I have tested my changes and ensured that all tests pass (`pdm run
test`).
- [x] I have formatted the code (`pdm run format`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have updated documentation where necessary.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Centralized, configurable timeout system for web interactions,
detection flows, publishing, and pagination.
* Optional retry with exponential backoff for operations that time out.

* **Improvements**
* Replaced fixed wait times with dynamic timeouts throughout workflows.
  * More informative timeout-related messages and diagnostics.

* **Tests**
* New and expanded test coverage for timeout behavior, pagination,
diagnostics, and retry logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Author: 1cu

CONTRIBUTING.md

@@ -187,6 +187,14 @@ All Python files must start with SPDX license headers:
 - Use appropriate log levels (DEBUG, INFO, WARNING, ERROR)
 - Log important state changes and decision points
 
+#### Timeout configuration
+- The default timeout (`timeouts.default`) already wraps all standard DOM helpers (`web_find`, `web_click`, etc.) via `WebScrapingMixin._timeout/_effective_timeout`. Use it unless a workflow clearly needs a different SLA.
+- Reserve `timeouts.quick_dom` for transient overlays (shipping dialogs, payment prompts, toast banners) that should render almost instantly; call `self._timeout("quick_dom")` in those spots to keep the UI responsive.
+- For single selectors that occasionally need more headroom, pass an inline override instead of creating a new config key, e.g. `custom = self._timeout(override = 12.5); await self.web_find(..., timeout = custom)`.
+- Use `_timeout()` when you just need the raw configured value (with optional override); use `_effective_timeout()` when you rely on the global multiplier and retry backoff for a given attempt (e.g. inside `_run_with_timeout_retries`).
+- Add a new timeout key only when a recurring workflow has its own timing profile (pagination, captcha detection, publishing confirmations, Chrome probes, etc.). Whenever you add one, extend `TimeoutConfig`, document it in the sample `timeouts:` block in `README.md`, and explain it in `docs/BROWSER_TROUBLESHOOTING.md`.
+- Encourage users to raise `timeouts.multiplier` when everything is slow, and override existing keys in `config.yaml` before introducing new ones. This keeps the configuration surface minimal.
+
 #### Examples
 ```python
 def parse_duration(text: str) -> timedelta:
@@ -297,4 +305,3 @@ See the [LICENSE.txt](LICENSE.txt) file for our project's licensing. All source
 - Use the translation system for all output—**never hardcode German or other languages** in the code.
 - If you add or change a user-facing message, update the translation file and ensure that translation completeness tests pass (`tests/unit/test_translations.py`).
 - Review the translation guidelines and patterns in the codebase for correct usage.
-

README.md

@@ -277,6 +277,27 @@ categories:
   Verschenken & Tauschen > Verleihen: 272/274
   Verschenken & Tauschen > Verschenken: 272/192
 
+# timeout tuning (optional)
+timeouts:
+  multiplier: 1.0                     # Scale all timeouts (e.g. 2.0 for slower networks)
+  default: 5.0                        # Base timeout for web_find/web_click/etc.
+  page_load: 15.0                     # Timeout for web_open page loads
+  captcha_detection: 2.0              # Timeout for captcha iframe detection
+  sms_verification: 4.0               # Timeout for SMS verification banners
+  gdpr_prompt: 10.0                   # Timeout when handling GDPR dialogs
+  publishing_result: 300.0            # Timeout for publishing status checks
+  publishing_confirmation: 20.0       # Timeout for publish confirmation redirect
+  pagination_initial: 10.0            # Timeout for first pagination lookup
+  pagination_follow_up: 5.0           # Timeout for subsequent pagination clicks
+  quick_dom: 2.0                      # Generic short DOM timeout (shipping dialogs, etc.)
+  update_check: 10.0                  # Timeout for GitHub update requests
+  chrome_remote_probe: 2.0            # Timeout for local remote-debugging probes
+  chrome_remote_debugging: 5.0        # Timeout for remote debugging API calls
+  chrome_binary_detection: 10.0       # Timeout for chrome --version subprocess
+  retry_enabled: true                 # Enables DOM retry/backoff when timeouts occur
+  retry_max_attempts: 2
+  retry_backoff_factor: 1.5
+
 # download configuration
 download:
   include_all_matching_shipping_options: false  # if true, all shipping options matching the package size will be included
@@ -329,6 +350,8 @@ login:
   password: ""
 ```
 
+Slow networks or sluggish remote browsers often just need a higher `timeouts.multiplier`, while truly problematic selectors can get explicit values directly under `timeouts`. Remember to regenerate the schemas after changing the configuration model so editors stay in sync.
+
 ### <a name="ad-config"></a>2) Ad configuration
 
 Each ad is described in a separate JSON or YAML file with prefix `ad_<filename>`. The prefix is configurable in config file.

docs/BROWSER_TROUBLESHOOTING.md

@@ -59,6 +59,18 @@ Please update your configuration to include --user-data-dir for remote debugging
 
 The bot will also provide specific instructions on how to fix your configuration.
 
+### Issue: Slow page loads or recurring TimeoutError
+
+**Symptoms:**
+- `_extract_category_from_ad_page` fails intermittently due to breadcrumb lookups timing out
+- Captcha/SMS/GDPR prompts appear right after a timeout
+- Requests to GitHub's API fail sporadically with timeout errors
+
+**Solutions:**
+1. Increase `timeouts.multiplier` in `config.yaml` (e.g. `2.0` doubles every timeout consistently).
+2. Override specific keys under `timeouts` (e.g. `pagination_initial: 20.0`) if only a single selector is problematic.
+3. Keep `retry_enabled` on so that DOM lookups are retried with exponential backoff.
+
 ## Common Issues and Solutions
 
 ### Issue 1: "Failed to connect to browser" with "root" error

schemas/config.schema.json

@@ -359,6 +359,137 @@
       "title": "PublishingConfig",
       "type": "object"
     },
+    "TimeoutConfig": {
+      "properties": {
+        "multiplier": {
+          "default": 1.0,
+          "description": "Global multiplier applied to all timeout values.",
+          "minimum": 0.1,
+          "title": "Multiplier",
+          "type": "number"
+        },
+        "default": {
+          "type": "number",
+          "minimum": 0.0,
+          "default": 5.0,
+          "description": "Baseline timeout for DOM interactions.",
+          "title": "Default"
+        },
+        "page_load": {
+          "default": 15.0,
+          "description": "Page load timeout for web_open.",
+          "minimum": 1.0,
+          "title": "Page Load",
+          "type": "number"
+        },
+        "captcha_detection": {
+          "default": 2.0,
+          "description": "Timeout for captcha iframe detection.",
+          "minimum": 0.1,
+          "title": "Captcha Detection",
+          "type": "number"
+        },
+        "sms_verification": {
+          "default": 4.0,
+          "description": "Timeout for SMS verification prompts.",
+          "minimum": 0.1,
+          "title": "Sms Verification",
+          "type": "number"
+        },
+        "gdpr_prompt": {
+          "default": 10.0,
+          "description": "Timeout for GDPR/consent dialogs.",
+          "minimum": 1.0,
+          "title": "Gdpr Prompt",
+          "type": "number"
+        },
+        "publishing_result": {
+          "default": 300.0,
+          "description": "Timeout for publishing result checks.",
+          "minimum": 10.0,
+          "title": "Publishing Result",
+          "type": "number"
+        },
+        "publishing_confirmation": {
+          "default": 20.0,
+          "description": "Timeout for publish confirmation redirect.",
+          "minimum": 1.0,
+          "title": "Publishing Confirmation",
+          "type": "number"
+        },
+        "pagination_initial": {
+          "default": 10.0,
+          "description": "Timeout for initial pagination lookup.",
+          "minimum": 1.0,
+          "title": "Pagination Initial",
+          "type": "number"
+        },
+        "pagination_follow_up": {
+          "default": 5.0,
+          "description": "Timeout for subsequent pagination navigation.",
+          "minimum": 1.0,
+          "title": "Pagination Follow Up",
+          "type": "number"
+        },
+        "quick_dom": {
+          "default": 2.0,
+          "description": "Generic short timeout for transient UI.",
+          "minimum": 0.1,
+          "title": "Quick Dom",
+          "type": "number"
+        },
+        "update_check": {
+          "default": 10.0,
+          "description": "Timeout for GitHub update checks.",
+          "minimum": 1.0,
+          "title": "Update Check",
+          "type": "number"
+        },
+        "chrome_remote_probe": {
+          "default": 2.0,
+          "description": "Timeout for local remote-debugging probes.",
+          "minimum": 0.1,
+          "title": "Chrome Remote Probe",
+          "type": "number"
+        },
+        "chrome_remote_debugging": {
+          "default": 5.0,
+          "description": "Timeout for remote debugging API calls.",
+          "minimum": 1.0,
+          "title": "Chrome Remote Debugging",
+          "type": "number"
+        },
+        "chrome_binary_detection": {
+          "default": 10.0,
+          "description": "Timeout for chrome --version subprocesses.",
+          "minimum": 1.0,
+          "title": "Chrome Binary Detection",
+          "type": "number"
+        },
+        "retry_enabled": {
+          "default": true,
+          "description": "Enable built-in retry/backoff for DOM operations.",
+          "title": "Retry Enabled",
+          "type": "boolean"
+        },
+        "retry_max_attempts": {
+          "default": 2,
+          "description": "Max retry attempts when retry is enabled.",
+          "minimum": 1,
+          "title": "Retry Max Attempts",
+          "type": "integer"
+        },
+        "retry_backoff_factor": {
+          "default": 1.5,
+          "description": "Exponential factor applied per retry attempt.",
+          "minimum": 1.0,
+          "title": "Retry Backoff Factor",
+          "type": "number"
+        }
+      },
+      "title": "TimeoutConfig",
+      "type": "object"
+    },
     "UpdateCheckConfig": {
       "description": "Configuration for update checking functionality.\n\nAttributes:\n    enabled: Whether update checking is enabled.\n    channel: Which release channel to check ('latest' for stable, 'preview' for prereleases).\n    interval: How often to check for updates (e.g. '7d', '1d').\n        If the interval is invalid, too short (<1d), or too long (>30d),\n        the bot will log a warning and use a default interval for this run:\n            - 1d for 'preview' channel\n            - 7d for 'latest' channel\n        The config file is not changed automatically; please fix your config to avoid repeated warnings.",
       "properties": {
@@ -428,6 +559,10 @@
     "update_check": {
       "$ref": "#/$defs/UpdateCheckConfig",
       "description": "Update check configuration"
+    },
+    "timeouts": {
+      "$ref": "#/$defs/TimeoutConfig",
+      "description": "Centralized timeout configuration."
     }
   },
   "title": "Config",

src/kleinanzeigen_bot/__init__.py

@@ -573,8 +573,9 @@ def load_ad(self, ad_cfg_orig:dict[str, Any]) -> Ad:
 
     async def check_and_wait_for_captcha(self, *, is_login_page:bool = True) -> None:
         try:
+            captcha_timeout = self._timeout("captcha_detection")
             await self.web_find(By.CSS_SELECTOR,
-                                "iframe[name^='a-'][src^='https://www.google.com/recaptcha/api2/anchor?']", timeout = 2)
+                                "iframe[name^='a-'][src^='https://www.google.com/recaptcha/api2/anchor?']", timeout = captcha_timeout)
 
             if not is_login_page and self.config.captcha.auto_restart:
                 LOG.warning("Captcha recognized - auto-restart enabled, abort run...")
@@ -624,7 +625,8 @@ async def fill_login_data_and_send(self) -> None:
 
     async def handle_after_login_logic(self) -> None:
         try:
-            await self.web_find(By.TEXT, "Wir haben dir gerade einen 6-stelligen Code für die Telefonnummer", timeout = 4)
+            sms_timeout = self._timeout("sms_verification")
+            await self.web_find(By.TEXT, "Wir haben dir gerade einen 6-stelligen Code für die Telefonnummer", timeout = sms_timeout)
             LOG.warning("############################################")
             LOG.warning("# Device verification message detected. Please follow the instruction displayed in the Browser.")
             LOG.warning("############################################")
@@ -634,9 +636,12 @@ async def handle_after_login_logic(self) -> None:
 
         try:
             LOG.info("Handling GDPR disclaimer...")
-            await self.web_find(By.ID, "gdpr-banner-accept", timeout = 10)
+            gdpr_timeout = self._timeout("gdpr_prompt")
+            await self.web_find(By.ID, "gdpr-banner-accept", timeout = gdpr_timeout)
             await self.web_click(By.ID, "gdpr-banner-cmp-button")
-            await self.web_click(By.XPATH, "//div[@id='ConsentManagementPage']//*//button//*[contains(., 'Alle ablehnen und fortfahren')]", timeout = 10)
+            await self.web_click(By.XPATH,
+                                 "//div[@id='ConsentManagementPage']//*//button//*[contains(., 'Alle ablehnen und fortfahren')]",
+                                 timeout = gdpr_timeout)
         except TimeoutError:
             pass
 
@@ -724,7 +729,8 @@ async def publish_ads(self, ad_cfgs:list[tuple[str, Ad, dict[str, Any]]]) -> Non
             count += 1
 
             await self.publish_ad(ad_file, ad_cfg, ad_cfg_orig, published_ads, AdUpdateStrategy.REPLACE)
-            await self.web_await(self.__check_publishing_result, timeout = 5 * 60)
+            publish_timeout = self._timeout("publishing_result")
+            await self.web_await(self.__check_publishing_result, timeout = publish_timeout)
 
             if self.config.publishing.delete_old_ads == "AFTER_PUBLISH" and not self.keep_old_ads:
                 await self.delete_ad(ad_cfg, published_ads, delete_old_ads_by_title = False)
@@ -924,7 +930,8 @@ async def publish_ad(self, ad_file:str, ad_cfg:Ad, ad_cfg_orig:dict[str, Any], p
         # wait for payment form if commercial account is used
         #############################
         try:
-            await self.web_find(By.ID, "myftr-shppngcrt-frm", timeout = 2)
+            short_timeout = self._timeout("quick_dom")
+            await self.web_find(By.ID, "myftr-shppngcrt-frm", timeout = short_timeout)
 
             LOG.warning("############################################")
             LOG.warning("# Payment form detected! Please proceed with payment.")
@@ -934,7 +941,8 @@ async def publish_ad(self, ad_file:str, ad_cfg:Ad, ad_cfg_orig:dict[str, Any], p
         except TimeoutError:
             pass
 
-        await self.web_await(lambda: "p-anzeige-aufgeben-bestaetigung.html?adId=" in self.page.url, timeout = 20)
+        confirmation_timeout = self._timeout("publishing_confirmation")
+        await self.web_await(lambda: "p-anzeige-aufgeben-bestaetigung.html?adId=" in self.page.url, timeout = confirmation_timeout)
 
         # extract the ad id from the URL's query parameter
         current_url_query_params = urllib_parse.parse_qs(urllib_parse.urlparse(self.page.url).query)
@@ -986,7 +994,8 @@ async def update_ads(self, ad_cfgs:list[tuple[str, Ad, dict[str, Any]]]) -> None
             count += 1
 
             await self.publish_ad(ad_file, ad_cfg, ad_cfg_orig, published_ads, AdUpdateStrategy.MODIFY)
-            await self.web_await(self.__check_publishing_result, timeout = 5 * 60)
+            publish_timeout = self._timeout("publishing_result")
+            await self.web_await(self.__check_publishing_result, timeout = publish_timeout)
 
         LOG.info("############################################")
         LOG.info("DONE: updated %s", pluralize("ad", count))
@@ -1080,6 +1089,7 @@ async def __set_special_attributes(self, ad_cfg:Ad) -> None:
             LOG.debug("Successfully set attribute field [%s] to [%s]...", special_attribute_key, special_attribute_value_str)
 
     async def __set_shipping(self, ad_cfg:Ad, mode:AdUpdateStrategy = AdUpdateStrategy.REPLACE) -> None:
+        short_timeout = self._timeout("quick_dom")
         if ad_cfg.shipping_type == "PICKUP":
             try:
                 await self.web_click(By.ID, "radio-pickup")
@@ -1091,7 +1101,7 @@ async def __set_shipping(self, ad_cfg:Ad, mode:AdUpdateStrategy = AdUpdateStrate
             if mode == AdUpdateStrategy.MODIFY:
                 try:
                     # when "Andere Versandmethoden" is not available, go back and start over new
-                    await self.web_find(By.XPATH, '//dialog//button[contains(., "Andere Versandmethoden")]', timeout = 2)
+                    await self.web_find(By.XPATH, '//dialog//button[contains(., "Andere Versandmethoden")]', timeout = short_timeout)
                 except TimeoutError:
                     await self.web_click(By.XPATH, '//dialog//button[contains(., "Zurück")]')
 
@@ -1120,7 +1130,7 @@ async def __set_shipping(self, ad_cfg:Ad, mode:AdUpdateStrategy = AdUpdateStrate
                         # (important for mode = UPDATE)
                         await self.web_find(By.XPATH,
                                             '//input[contains(@placeholder, "Versandkosten (optional)")]',
-                                            timeout = 2)
+                                            timeout = short_timeout)
                     except TimeoutError:
                         await self.web_click(By.XPATH, '//*[contains(@id, "INDIVIDUAL") and contains(@data-testid, "Individueller Versand")]')