Merge pull request #138 from matagus/last-fixes-before-1.0.0-version

Several fixes before version 1.0.0

Author: matagus

.gitignore

@@ -34,8 +34,9 @@ settings_local.py
 ##PyCharm
 .idea
 
-# Sphinx
+# Sphinx / mkdocs / readthedocs
 docs/_build
+site/
 
 # thumbnails, etc
 screenshots/.DS_Store

.readthedocs.yaml

@@ -12,7 +12,7 @@ build:
 
 mkdocs:
   configuration: mkdocs.yml
-  fail_on_warning: false
+  fail_on_warning: true
 
 # Optional but recommended, declare the Python requirements required
 # to build your documentation

README.md

@@ -97,9 +97,25 @@ PLANET = {
     "USER_AGENT": "MyPlanet/1.0",  # Customize the User-Agent for feed requests
     "RECENT_POSTS_LIMIT": 10,  # Number of recent posts to show
     "RECENT_BLOGS_LIMIT": 10,  # Number of recent blogs to show
+    "FETCH_ORIGINAL_CONTENT": False,  # Fetch and archive full post content from the original URL
+    "FETCH_CONTENT_DELAY": 0,  # Seconds to wait between content fetches (int or float)
 }
 ```
 
+#### Original Content Archiving
+
+When `FETCH_ORIGINAL_CONTENT` is `True`, django-planet will fetch the full HTML of each post's original URL using `readability-lxml` to extract the article body. The result is stored in `Post.original_content` and shown on the post detail page instead of the feed summary.
+
+```python
+PLANET = {
+    "FETCH_ORIGINAL_CONTENT": True,
+    "FETCH_CONTENT_DELAY": 1,  # 1 second between fetches to be polite to servers
+}
+```
+
+- If fetching fails for a post, a WARNING is logged and `original_content` remains `None` (the feed summary is shown as fallback).
+- Use the `planet_fetch_post_content` management command to backfill existing posts.
+
 4. Run migrations:
 
 ```bash
@@ -323,6 +339,90 @@ All admin interfaces include search and filtering capabilities.
 - Updates feed metadata (etag, last_checked)
 - Creates new Post and Author entries as needed
 
+**`planet_fetch_post_content`**
+- Backfills `original_content` for posts where it is missing
+- Optional `--feed <id>` argument to limit to a specific feed
+- Optional `--limit <n>` argument to cap the number of posts processed
+- Respects `FETCH_CONTENT_DELAY` between requests
+
+## 🔍 Post Filtering
+
+By default, all feed entries are saved. You can configure a **post filter backend** to accept only relevant posts before they are stored.
+
+### Configuration
+
+```python
+PLANET = {
+    "POST_FILTER_BACKEND": "planet.backends.accept_all.AcceptAllBackend",  # default
+    "TOPIC_KEYWORDS": [],
+}
+```
+
+### Built-in Backends
+
+**`planet.backends.accept_all.AcceptAllBackend`** *(default)*
+Accepts every entry unchanged. No configuration required.
+
+**`planet.backends.keyword.KeywordFilterBackend`**
+Accepts entries whose title or summary contains at least one of the configured keywords (case-insensitive). Rejected entries are logged at `INFO` level.
+
+```python
+PLANET = {
+    "POST_FILTER_BACKEND": "planet.backends.keyword.KeywordFilterBackend",
+    "TOPIC_KEYWORDS": ["python", "django", "open source"],
+}
+```
+
+When `TOPIC_KEYWORDS` is empty the backend accepts all entries (fail-open).
+
+### Writing a Custom Backend
+
+Subclass `BasePostFilterBackend` and implement `filter_entries`:
+
+```python
+from planet.backends.base import BasePostFilterBackend
+
+class MyBackend(BasePostFilterBackend):
+    def filter_entries(self, entries, feed):
+        # entries: list of feedparser entry objects
+        # feed: planet.models.Feed instance
+        return [e for e in entries if passes_my_check(e)]
+```
+
+Then point to it in your settings:
+
+```python
+PLANET = {
+    "POST_FILTER_BACKEND": "myapp.backends.MyBackend",
+}
+```
+
+## 📋 Logging
+
+django-planet uses Python's standard `logging` module. All loggers use names under the `planet.*` namespace (e.g. `planet.utils`, `planet.management.commands.planet_update_all_feeds`).
+
+Following Python library best practices, **no handlers are attached by default** — the host project controls all logging output. Add a `LOGGING` configuration in your Django settings to see log output:
+
+```python
+LOGGING = {
+    "version": 1,
+    "disable_existing_loggers": False,
+    "handlers": {
+        "console": {
+            "class": "logging.StreamHandler",
+        },
+    },
+    "loggers": {
+        "planet": {
+            "handlers": ["console"],
+            "level": "INFO",  # Use "DEBUG" for more verbosity
+        },
+    },
+}
+```
+
+At `INFO` level you'll see feed add/update summaries and 304 skips. At `DEBUG` level you'll also see individual fetch details, per-entry creation, and `to_datetime()` edge cases.
+
 ## 📸 Screenshots
 
 ### Post List

docs/admin.md

@@ -0,0 +1,53 @@
+# Admin Interface
+
+Django-planet registers all models in the Django admin with search, filtering, and cross-linking between related objects.
+
+## "Add Feed by URL" Workflow
+
+The Feed admin has a special workflow for adding new feeds. When you click "Add Feed", you only need to provide the feed URL:
+
+1. Enter the feed URL and click Save
+2. django-planet automatically creates a **Blog** entry (using the feed's domain as a placeholder title) if one doesn't already exist
+3. A **Feed** stub is created, ready to be populated on the next `planet_update_all_feeds` run
+
+This is the same operation as `python manage.py planet_add_feed <url>`, but accessible from the admin.
+
+!!! note
+
+    The feed's title and metadata will be populated automatically the next time feeds are updated.
+
+## BlogAdmin
+
+- **List display:** title, URL, date created
+- **Search:** by title, URL
+- **Inline feeds:** Read-only tabular inline showing all feeds for the blog, with links to each feed's admin page
+
+## FeedAdmin
+
+- **List display:** title, URL, blog, language, etag, last modified, last checked, active status
+- **List filter:** by language
+- **Search:** by title, URL, blog title
+- **Fieldsets:**
+    - *General:* title, URL, blog, language
+    - *Feed Status:* etag, last modified, last checked, is_active
+    - *Authors:* read-only list of all authors who have posts in this feed, with links to each author's admin page
+
+## PostAdmin
+
+- **List display:** title, feed, guid, date published, date created
+- **List filter:** by feed title, language
+- **Search:** by title, blog title
+- **Optimized queries:** uses `select_related` for feed and blog to minimize database queries
+
+## AuthorAdmin
+
+- **List display:** name, email
+- **Search:** by name
+- **Fieldsets:**
+    - *General:* name, email, profile URL
+    - *Feeds:* read-only list of all feeds this author has contributed to, with links to each feed's admin page
+
+## PostAuthorDataAdmin
+
+- **List display:** author name, is_contributor flag, post
+- **List filter:** by is_contributor, author

docs/configuration.md

@@ -0,0 +1,137 @@
+# Configuration Reference
+
+All django-planet settings are defined in a single `PLANET` dictionary in your Django settings module. Any key you don't specify falls back to its default.
+
+## PLANET_CONFIG
+
+```python
+PLANET = {
+    "USER_AGENT": "MyPlanet/1.0",
+    "RECENT_POSTS_LIMIT": 10,
+    "RECENT_BLOGS_LIMIT": 10,
+    "POST_FILTER_BACKEND": "planet.backends.accept_all.AcceptAllBackend",
+    "TOPIC_KEYWORDS": [],
+    "FETCH_ORIGINAL_CONTENT": False,
+    "FETCH_CONTENT_DELAY": 0,
+}
+```
+
+### Settings Reference
+
+`USER_AGENT`
+:   **Type:** `str`
+    **Default:** `"Django Planet/<version>"`
+
+    The User-Agent header sent when fetching feeds and post content.
+
+`RECENT_POSTS_LIMIT`
+:   **Type:** `int`
+    **Default:** `10`
+
+    Number of posts shown by the `{% recent_posts %}` template tag.
+
+`RECENT_BLOGS_LIMIT`
+:   **Type:** `int`
+    **Default:** `10`
+
+    Number of blogs shown by the `{% recent_blogs %}` template tag.
+
+`POST_FILTER_BACKEND`
+:   **Type:** `str` (dotted Python path)
+    **Default:** `"planet.backends.accept_all.AcceptAllBackend"`
+
+    The backend class used to filter incoming feed entries before they are saved. See [Post Filter Backends](#post-filter-backends) below.
+
+`TOPIC_KEYWORDS`
+:   **Type:** `list[str]`
+    **Default:** `[]`
+
+    Keywords used by `KeywordFilterBackend` to filter posts. Only entries whose title or summary contains at least one keyword (case-insensitive) are accepted.
+
+`FETCH_ORIGINAL_CONTENT`
+:   **Type:** `bool`
+    **Default:** `False`
+
+    When `True`, django-planet fetches the full HTML of each post's original URL using `readability-lxml` and stores it in `Post.original_content`. See [Usage > Content Archiving](usage.md#original-content-archiving).
+
+`FETCH_CONTENT_DELAY`
+:   **Type:** `int | float`
+    **Default:** `0`
+
+    Seconds to wait between content fetches. Set this to a positive value (e.g., `1`) to be polite to origin servers.
+
+## Post Filter Backends
+
+By default, all feed entries are saved. You can configure a post filter backend to accept only relevant posts before they are stored.
+
+### AcceptAllBackend (default)
+
+```python
+PLANET = {
+    "POST_FILTER_BACKEND": "planet.backends.accept_all.AcceptAllBackend",
+}
+```
+
+Accepts every entry unchanged. No configuration required.
+
+### KeywordFilterBackend
+
+```python
+PLANET = {
+    "POST_FILTER_BACKEND": "planet.backends.keyword.KeywordFilterBackend",
+    "TOPIC_KEYWORDS": ["python", "django", "open source"],
+}
+```
+
+Accepts entries whose title or summary contains at least one of the configured keywords (case-insensitive). Rejected entries are logged at `INFO` level.
+
+When `TOPIC_KEYWORDS` is empty, the backend accepts all entries (fail-open).
+
+### Writing a Custom Backend
+
+Subclass `BasePostFilterBackend` and implement `filter_entries`:
+
+```python
+from planet.backends.base import BasePostFilterBackend
+
+
+class MyBackend(BasePostFilterBackend):
+    def filter_entries(self, entries, feed):
+        # entries: list of feedparser entry objects
+        # feed: planet.models.Feed instance
+        return [e for e in entries if passes_my_check(e)]
+```
+
+Then point to it in your settings:
+
+```python
+PLANET = {
+    "POST_FILTER_BACKEND": "myapp.backends.MyBackend",
+}
+```
+
+## Logging
+
+django-planet uses Python's standard `logging` module. All loggers use names under the `planet.*` namespace (e.g., `planet.utils`, `planet.management.commands.planet_update_all_feeds`).
+
+Following Python library best practices, **no handlers are attached by default** — the host project controls all logging output. Add a `LOGGING` configuration to see log output:
+
+```python
+LOGGING = {
+    "version": 1,
+    "disable_existing_loggers": False,
+    "handlers": {
+        "console": {
+            "class": "logging.StreamHandler",
+        },
+    },
+    "loggers": {
+        "planet": {
+            "handlers": ["console"],
+            "level": "INFO",  # Use "DEBUG" for more verbosity
+        },
+    },
+}
+```
+
+At `INFO` level you'll see feed add/update summaries and 304 skips. At `DEBUG` level you'll also see individual fetch details, per-entry creation, and `to_datetime()` edge cases.