Session delays

[Gallaecio]

See CHANGES.rst for an overview of the changes.

The changes also include a refactoring of session tests, mostly a split of them into ~20 files of less than 500 lines each, extracting stats expectation checks into a helper function, and the merging of a few functions to use parametrize instead. I would not bother reviewing every test file, I recommend focusing on test_sessions_pool, which contains the tests for the main changes.

cc @proway2 @gutsytechster

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.50%. Comparing base (5b9cd49) to head (178360f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #262      +/-   ##
==========================================
+ Coverage   97.36%   97.50%   +0.13%     
==========================================
  Files          15       15              
  Lines        1939     2006      +67     
  Branches      356      365       +9     
==========================================
+ Hits         1888     1956      +68     
  Misses         25       25              
+ Partials       26       25       -1     

Files with missing lines	Coverage Δ
scrapy_zyte_api/_session.py	99.60% <100.00%> (+0.29%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR introduces configurable delays between reuses of plugin-managed sessions (optionally randomized), adds per-pool configuration (including dict-based pool specs), updates docs/terminology, and refactors + expands session test coverage.

Changes:

• Add ZYTE_API_SESSION_DELAY + ZYTE_API_SESSION_RANDOMIZE_DELAY, and implement per-session reuse scheduling in the session rotation queue.
• Introduce ZYTE_API_SESSION_POOLS (and deprecate ZYTE_API_SESSION_POOL_SIZES), allowing per-pool delay/size/randomize_delay overrides and dict-based pool specs from SessionConfig.pool / request meta.
• Split/refactor session tests into many smaller files and add helpers for session stats assertions; update docs and changelog accordingly.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tox.ini	Pins typing_extensions for min environments; updates x402-related deps.
pyproject.toml	Adds runtime dependency on typing_extensions; constrains optional x402.
scrapy_zyte_api/_session.py	Implements session reuse delays, per-pool configs, improved PoolError messaging, new troubleshooting URL helper.
tests/init.py	Introduces shared UNSET, SESSION_SETTINGS, and a get_session_stats helper.
tests/helpers.py	Adds assert_session_stats normalizing helper for stats assertions.
tests/conftest.py	Enables assert rewriting for tests.helpers.
tests/test_api_requests.py	Uses shared UNSET instead of local definition.
tests/test_addon.py	Adds addon/manual settings parity test for session retry policy settings.
tests/test_request_fingerprinter.py	Updates terminology in a comment.
tests/test_sessions_utils.py	Adds unit test coverage for is_session_init_request.
tests/test_sessions_refresh.py	Adds coverage for refresh behavior and concurrent refresh scenarios.
tests/test_sessions_queue.py	Adds coverage for queue empty behavior and max-attempts handling.
tests/test_sessions_pool.py	Adds broad coverage for pool IDs, pool params caching, delays, randomization, and pool sizes/deprecation logging.
tests/test_sessions_poet.py	Adds provider integration coverage with scrapy_poet.
tests/test_sessions_max.py	Adds coverage for max bad inits/check failures/errors behavior.
tests/test_sessions_init_url.py	Adds coverage for session init URL precedence (params override vs request URL).
tests/test_sessions_init_precedence.py	Adds exhaustive coverage for params/location precedence across settings + request meta and session configs.
tests/test_sessions_init_location_config.py	Adds coverage for LocationSessionConfig location_* hooks and registry reset safety.
tests/test_sessions_init_location.py	Adds coverage for SessionConfig.location override behaviors and precedence pitfalls.
tests/test_sessions_init_bad.py	Adds coverage for ZYTE_API_SESSION_MAX_BAD_INITS_PER_POOL.
tests/test_sessions_errors.py	Adds coverage for retry stop conditions, missing session IDs, and exception mapping into stats/reasons.
tests/test_sessions_enabled.py	Adds coverage for enabling/disabling sessions via setting/meta and via SessionConfig.enabled.
tests/test_sessions_cookies.py	Adds coverage for cookie behavior with/without sessions and dont_merge_cookies.
tests/test_sessions_config.py	Adds coverage for @session_config behaviors including process_request variants and params errors.
tests/test_sessions_check_errors.py	Adds coverage for checker precedence over max errors.
tests/test_sessions_check_default.py	Adds coverage for default checker’s location-based behavior and unsupported setLocation.
tests/test_sessions_check_custom.py	Adds coverage for custom checker variants, CloseSpider, exceptions, and meta propagation into init requests.
tests/test_sessions_assign.py	Adds coverage for session assignment meta key selection (zyte_api vs automap) and provider consistency.
docs/usage/session.rst	Renames/retunes session terminology; adds troubleshooting section; documents new delay/pooled settings.
docs/reference/settings.rst	Documents new ZYTE_API_SESSION_DELAY, ZYTE_API_SESSION_POOLS, ZYTE_API_SESSION_RANDOMIZE_DELAY; updates wording.
docs/reference/meta.rst	Updates wording for session-enabled and session-pool meta keys (but see suggestion).
CHANGES.rst	Adds 0.33.0 notes covering delays, dict-based pools, deprecation, terminology, troubleshooting section.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wRAR]

(this now has conflicts, which probably was expected)