Skip to content

docs: add AGENTS.md and supplementary developer guides#312

Closed
rmax wants to merge 1 commit into
feat/multi-key-priority-start-urlsfrom
chore/agents-md
Closed

docs: add AGENTS.md and supplementary developer guides#312
rmax wants to merge 1 commit into
feat/multi-key-priority-start-urlsfrom
chore/agents-md

Conversation

@rmax

@rmax rmax commented May 18, 2026

Copy link
Copy Markdown
Owner

Summary

Adds comprehensive developer/agent guide to the repository, covering everything needed for efficient contribution:

AGENTS.md — Main guide at repo root covering:

  • Project architecture (RedisMixin, RedisSpider, RedisCrawlSpider, Scheduler, etc.)
  • Multi-key mode documentation (PR feat: add optional multi-key priority support for RedisSpider #311 feature)
  • Quick start, testing with FakeRedisServer, tox matrix
  • Pre-commit hooks, Sphinx docs build
  • CI/CD workflows and debugging
  • PR workflow (branch naming, commit conventions, co-author credit, squash merge)
  • Security practices and dependency management
  • Links to supplementary docs

docs/coding-standards.rst — Python coding standards:

  • Style guide (PEP 8, 120 chars), linting config
  • Type hints for public APIs, NumPy/Google docstrings
  • Best practices (isinstance guards, backward compatibility, logging, context managers)

docs/security-guide.rst — Security guide:

  • Bandit scanning, Dependabot, dependency security
  • Redis TLS/SSL and authentication configuration
  • Input validation and request fingerprinting
  • Known concerns (pickle serializer, Redis exposure, multi-tenant isolation)

Closes #310

Add comprehensive developer/agent guide with:
- AGENTS.md: project overview, architecture, quick start,
  coding conventions, testing, CI/CD, PR workflow, security
- docs/coding-standards.rst: Python style, type hints, docstrings,
  best practices for the project
- docs/security-guide.rst: bandit, Redis connection security,
  input validation, dependency security

Co-authored-by: Masoud Lakestani <46046899+MasoudLakestani@users.noreply.github.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 40b004d945

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +99 to +102
try:
self.redis_key_check_interval = int(self.redis_key_check_interval)
except (TypeError, ValueError):
raise ValueError("redis_key_check_interval must be an integer or None")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject negative redis_key_check_interval values

The interval parser accepts negative numbers (e.g. -1) and stores them as-is, but _maybe_check_priority_scan() treats any truthy interval as enabled and checks current_time - last_scan >= interval, which is always true for negative values. In multi-key mode this forces a priority scan on every polling cycle, causing immediate preemption and extra Redis cardinality calls instead of the documented periodic behavior; a mistaken negative setting should be rejected up front.

Useful? React with 👍 / 👎.

@rmax rmax changed the base branch from master to feat/multi-key-priority-start-urls May 18, 2026 21:50
@rmax rmax deleted the branch feat/multi-key-priority-start-urls May 19, 2026 14:03
@rmax rmax closed this May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add optional support for prioritized multiple Redis input keys in RedisSpider

1 participant