Open
Conversation
Merges PR mediacms-io#1226 logging improvements. Adds structured exception logging to all exception handlers (tasks, views, helpers, models) while preserving existing exception handling behavior. Enhanced logging configuration with console handler and formatters available when DEBUG=True. Includes comprehensive documentation for administrators and developers. - Preserves ERROR-level file logging as default (production-safe) - Adds logging to 9 files: tasks.py, backends.py, helpers.py, methods.py, models/media.py, uploader/views.py, users/views.py, manage.py, settings.py - Adds admin docs (Section 29) and developer docs (Section 8)
- Fix logger initialization order in settings.py - Add app-specific loggers while preserving ERROR-level file logging - Replace generic exception messages with contextual logging - Add operational logging for encoding, uploads, and media lifecycle - Standardize exception handling with specific types and logger.exception() - Create logging_utils module with helpers for structured logging All exception handlers now include relevant context (IDs, file paths, operation details) to improve debugging and operational monitoring.
1. Enhanced logging configuration (`cms/settings.py`) * Improved formatters to include function name and line number * Added separate detailed formatter for console output in DEBUG mode * Enhanced DEBUG mode with more detailed information 2. Fixed missing logging * `saml_auth/adapter.py`: Replaced basic logging.error() calls with proper logger calls including context; added logging for SAML authentication and role mapping * `files/tasks.py`: Fixed bare `except:` clause in `media_init` task with proper exception handling and logging * `cms/middleware.py`: Added logging for approval middleware blocking actions * `files/helpers.py`: Added logging for file operations (create, delete, move, command execution) * `files/methods.py`: Added logging for permission checks, state transitions, and notifications * `uploader/views.py`: Enhanced upload logging with file details, size, type, and error context 3. Standardized log messages * Converted f-string log messages to structured format with key=value pairs * Ensured consistent format across all log messages * Added required context variables (user_id, friendly_token, encoding_id, etc.) 4. Aligned severity levels * Changed routine operations (like thumbnail updates) from INFO to DEBUG * Ensured encoding failures are logged as ERROR * Properly categorized warnings vs errors 5. Added DEBUG mode enhancements * Added function entry/exit logging in key operations * Added intermediate value logging in complex operations * Added detailed object state logging for debugging 6. Removed duplicative logging * Reviewed and optimized logging statements
- Add ENABLE_SQL_DEBUG_LOGGING setting (default: False) to control SQL query logging - SQL queries are now opt-in even when DEBUG=True to reduce log noise - Set celery logger to INFO level to reduce noise while keeping celery.task at DEBUG - Suppress useless celery.utils.functional debug messages at WARNING level - Update admin and developer documentation with new configuration options
- Add logging to media deletion API view and post_delete signal handler - Explicitly set django.db.backends logger to ERROR when SQL debug logging is disabled - Ensures deletion events are logged and SQL queries remain suppressed by default
…I operations - Add login success/failure logging with IP addresses - Log user creation, deletion, password changes, and approval actions - Add role/permission change logging (including SAML role mapping) - Log media updates and all bulk operations (state changes, permissions, etc.) - Add admin management action logging (media, comments, users deletion) - Use INFO level for auditable events, WARNING for failed auth attempts
- Add logging to user deletion signal with content counts - Add logging to subtitle creation/update signals - Add logging to encoding creation/update/deletion signals - Add logging to video chapter deletion signal - Add logging to media category m2m_changed signal - Add logging to RBAC group category m2m_changed signal - Add django-allauth signal handlers for password reset, email confirmation, password change, and account signup All signal handlers use structured logging with key-value pairs for easy parsing and analysis.
…nd signal handler for consistency with django-allauth updates.
…itialization - Move error handling from logging to business logic in post_user_create - Remove unnecessary try/except blocks around logging in signal handlers - Remove redundant logger initialization, use module-level logger - Add error handling for database operations in delete_content
…eivers - Add post_save/post_delete signal handlers with logging for Category, Tag, Comment, Playlist, Rating, Page, License, Channel, MediaAction, Identity Provider, and SAML models - Use INFO level for creation/deletion, DEBUG for updates - Follow structured logging patterns with key-value pairs - Fix flake8 errors (unused variables, import ordering)
Add default values for CAN_EDIT, CAN_EDIT_MEDIA, CAN_DELETE_MEDIA, CAN_DELETE_COMMENTS, CAN_DELETE, media_object, and current_site in context processor. Fix templates to conditionally access optional variables. This prevents ValueError/AttributeError/TypeError messages from flooding debug logs when templates reference variables that aren't always available.
Remove references to non-existent media_object.summary attribute in media.html template. Fix crispy_custom_field.html to safely check field/widget types without accessing non-existent attributes (input_type on Textarea, choices on BooleanField). Prevents debug log flooding from template resolution errors.
Contributor
Author
|
This is progressing quite nicely. There are a number of rendering problems I have encountered during testing that I'm going to include here due to the fact that they are VERY noisy. To get an idea of what it's looking like, this is the |
Enhance all login paths (API, django-allauth, SAML) to capture and log failed login attempts with consistent format. Distinguish between user_not_found, wrong_password, user_deactivated, and user_not_approved failure types. Include IP addresses in all failed login logs for security monitoring.
Contributor
Author
|
After inspection of the logs, I have noticed that in my setup, IP addresses are not always properly rendered. This is likely related to #1204 and suspect that my branch over there will resolve it. I'd like to leave the logic in place here and consider pulling that fix here as well. I've added a series of signals/receivers to capture most Django events (like creates, login fails, etc), and they have already been useful for other efforts I have in flight. |
Implement proxy-aware middleware and IP extraction utilities to correctly handle client IP addresses and sessions when MediaCMS is deployed behind reverse proxies, load balancers, or other HTTP proxies. Features: - Proxy-aware middleware that extracts real client IP from X-Forwarded-For - Trusted proxy validation to prevent IP spoofing attacks - Support for multiple proxy hops and CIDR network notation - Configurable settings with safe defaults for backward compatibility - Comprehensive documentation with examples for common scenarios Fixes mediacms-io#1204
- Remove USE_X_FORWARDED_PORT and USE_X_FORWARDED_PREFIX (unused) - Change TRUSTED_PROXIES default from example values to empty list - Improve inline documentation for TRUSTED_PROXIES setting
Contributor
Author
|
login failure w/o reverse proxy (note, I suspect I can't see the real IP due to docker on Mac networking limitations, all requests on my Mac always show the docker network gateway as client ip 192.168.65.1; this is the same behavior before and after this change) login failure w/reverse proxy successful login w/o reverse proxy (note same caveat as above) successful login w/reverse proxy* |
Contributor
Author
|
Alright, put a fork in this one, I'm calling it done. I've got most, if not all, the important stuff properly logging now. I kept existing behavior the same, with the exception of client ip detection w/reverse proxies -- it should log the real client IP based on X-FORWARDED-FOR and/or X-REAL-IP. There were also a number of rendering errors related to model fields having no default value for frontend rendering (e.g. IS_MANAGER, IS_EDITOR...). I gave them sensible (False) defaults. Implemented a "TRUSTED_PROXIES" configuration that will enable the behavior if the proxy is actually trusted. I can provide additional log information or details as needed. I've run through about 50 tests, uploads, tags, bulk actions, etc. All are properly logged and I now have a better understanding of how things work in a typical flow! I do seem to have problems WITHOUT a reverse proxy but think it caused by Docker on Mac networking limitations -- it will likely need validation in a closer to real linux environment to be sure it's grabbing the correct remote_addr / client ip. If you do NOT specify DEBUG=True, log levels remain at ERROR and up, however there WILL be a few new ones that were not logged properly, if at all. I left examples for the reverse proxy setup commented in the local_settings, nginx and uwsgi configs. Finally, I added a slew of documentation around the use for both administrators and developers. Needs validated. Of note, a section added for SAML troubleshooting, which should now properly deliver error messages to make it easier when configuring or bug hunting. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR refreshes #1226 against latest main branch. Better late than never.
This enhances logging across the codebase to improve debugging, monitoring, and operational visibility. It introduces structured logging utilities, adds contextual exception logging throughout the application, and implements a unified log level configuration mechanism.
Key Changes:
New Logging Utilities Module (
cms/logging_utils.py):log_operation_timing()- Context manager for timing operations with automatic start/complete/failure logginglog_exception_with_context()- Helper for logging exceptions with additional contextsanitize_path()- Utility to truncate long file paths in logssanitize_token()- Utility to mask sensitive tokens/keys in logsEnhanced Logging Configuration (
cms/settings.py):LOGLEVELvariable now controls log levels across all MediaCMS componentsLOGLEVELcan be set via environment variable (LOGLEVEL=INFO) or overridden inlocal_settings.pyDEBUG=True, log level is automatically set toDEBUGfor comprehensive development loggingComprehensive Exception Logging:
logger.exception()with specific exception typesOperational Logging:
Documentation:
docs/admins_docs.mdSection 29):celery_long.log,celery_short.log,celery_beat.log)docs/developers_docs.mdSection 8)Files Modified:
cms/logging_utils.py(new file)cms/settings.pyfiles/tasks.pyfiles/backends.pyfiles/helpers.pyfiles/methods.pyfiles/models/media.pyuploader/views.pyusers/views.pydocs/admins_docs.mddocs/developers_docs.mdSteps
Pre-deploy
Post-deploy