Skip to content

DEVELOPER.md

Latest commit

 

History

History
621 lines (469 loc) · 23.1 KB

DEVELOPER.md

File metadata and controls

621 lines (469 loc) · 23.1 KB

DEVELOPER.md

Welcome to the Developer Guide for QuestByCycle, a gamified bicycling platform. This document provides an in-depth understanding of the architecture, codebase, and development practices. Whether you're adding new features, fixing bugs, or maintaining the system, this guide will help you navigate the code and understand its intricacies.

Table of Contents

  1. Project Structure
  2. Setting Up the Development Environment
  3. Key Components
  4. Admin Functionality
  5. User Functionality
  6. AI Quest Generation
  7. Static Assets
  8. Testing and Debugging
  9. Deployment

Project Structure

Directory Overview

The project is organized as follows:

  • app/: The main application directory.
    • static/: Holds compiled assets and other static files.
      • dist/: Bundled JavaScript and CSS generated by Vite.
      • scss/: Source stylesheets compiled by Vite.
      • icons/, images/, qr_codes/, videos/: Media assets.
      • webfonts/: Font files.
    • templates/: Contains HTML templates for rendering views.
      • modals/: Templates for modal dialogs.
    • __init__.py: Initializes the Flask application.
    • admin.py: Admin-specific routes and logic.
    • ai.py: AI quest generation logic.
    • auth.py: Authentication routes and logic.
    • badges.py: Badge management logic.
    • config.py: Configuration settings.
    • forms.py: WTForms definitions.
    • games.py: Game management logic.
    • main.py: Main routes and views.
    • models.py: SQLAlchemy models.
    • notifications.py: Email helpers and Web Push utilities.
    • paypal.py: PayPal API integration.
    • profile.py: User profile management logic.
    • push.py: Endpoints for managing push subscriptions.
    • quests.py: Quest management and submission logic.
    • scheduler.py: Schedules recurring jobs.
    • social.py: Social media integration logic.
    • tasks.py: Background job functions executed by RQ workers.
    • utils.py: Utility functions.
    • activitypub_utils.py, constants.py, decorators.py, webfinger.py: Supporting modules.
  • csv/: Contains CSV files for bulk data import.
  • docs/: Documentation files.
  • frontend/: Vite-based JavaScript and SCSS source files.
  • migrations/: Database migration scripts.
  • .gitignore: Git ignore file.
  • .env: Environment configuration file. Set PAYPAL_CLIENT_ID, PAYPAL_CLIENT_SECRET, and PAYPAL_API_BASE for PayPal integration.
  • LICENSE.md: License information.
  • README.md: Project overview and setup instructions.
  • pyproject.toml: Project configuration and Python dependencies managed by Poetry.
  • wsgi.py: WSGI entry point for deploying the application.

Important Files

  • app/__init__.py: Provides the create_app factory, loads configuration via load_config, overrides url_for for testing, initializes extensions, and registers blueprints.
  • app/models.py: Defines the database models.
  • app/forms.py: Defines the forms used in the application.
  • app/config.py: Centralized configuration loaded from environment variables.
  • app/notifications.py: Email helpers and Web Push utilities.
  • app/push.py: Push notification subscription and delivery endpoints.
  • app/scheduler.py: Schedules recurring maintenance jobs.
  • app/tasks.py: Background jobs executed by RQ workers.
  • app/paypal.py: PayPal API integration.
  • app/utils/: Package of utility modules used across the application.
  • app/templates/: Contains HTML templates for rendering views.
  • pyproject.toml: Lists the dependencies required for the project and is managed by Poetry.

Setting Up the Development Environment

Prerequisites

Ensure you have the following installed:

  • Python 3.11
  • PostgreSQL
  • Redis
  • Node.js 22 and npm
  • ffmpeg (optional for video compression)
  • Poetry (with a Python 3.11 virtualenv)

Installation

  1. Clone the repository: ```bash git clone https://github.com/your-repo/your-project.git cd your-project ```

  2. Install Python dependencies (use Python 3.11): ```bash poetry env use 3.11 poetry install --sync ```

  3. Install frontend dependencies and build assets: ```bash npm install npm run build ```

  4. Set up the database: Create a PostgreSQL database and update the .env file with your database credentials.

  5. Database (development): This project initializes tables automatically on first run; if you use migrations, run them with your preferred tool. Otherwise proceed to running the app.

Configuration

Update the .env file with settings appropriate for your environment. The application reads the following variables:

Main

  • UPLOAD_FOLDER: Directory used for user uploads.
  • VERIFICATIONS: Directory for verification images.
  • BADGE_IMAGE_DIR: Directory for badge graphics.
  • TASKCSV: Directory containing bulk quest CSV files.
  • LOCAL_DOMAIN: Base domain for generating absolute URLs.
  • FFMPEG_PATH: Path to the ffmpeg executable for video processing.
  • PLACEHOLDER_IMAGE: Default image path used when no image is provided.
  • SQLALCHEMY_ECHO: Set to true to log SQL statements.
  • ASSET_VERSION: Cache-busting string appended to static asset URLs.

Google Cloud Storage

  • GCS_BUCKET: Name of the bucket used for uploads.
  • GCS_BASE_URL: Base URL for serving uploaded media.
  • GCS_STORAGE_CLASS: Storage class for uploaded files (ARCHIVE is cheapest).

Security and Sessions

  • DEFAULT_SUPER_ADMIN_USERNAME: Username for the initial super admin account.
  • DEFAULT_SUPER_ADMIN_PASSWORD: Password for the initial super admin.
  • DEFAULT_SUPER_ADMIN_EMAIL: Email for the initial super admin.
  • SECRET_KEY: Secret used to sign session cookies.
  • DATA_ENCRYPTION_KEY: Fernet key used to encrypt sensitive per-game credentials; falls back to data_encryption.key when unset.
  • SESSION_COOKIE_SECURE: Require HTTPS for the session cookie.
  • SESSION_COOKIE_NAME: Name of the session cookie.
  • SESSION_COOKIE_SAMESITE: SameSite policy for the session cookie.
  • SESSION_COOKIE_DOMAIN: Domain attribute for the session cookie, if needed.
  • SESSION_REFRESH_EACH_REQUEST: Refresh session on every request when true.
  • REMEMBER_COOKIE_DURATION_DAYS: Lifetime of the remember-me cookie in days.

Flask

  • DEBUG: Enables Flask debug mode.
  • SQLALCHEMY_DATABASE_URI: Database connection string.

OpenAI

  • OPENAI_API_KEY: API key used for AI quest generation.

Mail

  • MAIL_SERVER: Hostname of the mail server.
  • MAIL_PORT: Port for the mail server.
  • MAIL_USE_TLS: Enable TLS for outgoing mail.
  • MAIL_USE_SSL: Enable SSL for outgoing mail.
  • MAIL_USERNAME: Username for the mail server.
  • MAIL_PASSWORD: Password for the mail server.
  • MAIL_DEFAULT_SENDER: Default address used as the sender.

Social Media

  • TWITTER_USERNAME: Twitter handle used for updates.
  • TWITTER_API_KEY: Twitter API key.
  • TWITTER_API_SECRET: Twitter API secret.
  • TWITTER_ACCESS_TOKEN: Twitter access token.
  • TWITTER_ACCESS_TOKEN_SECRET: Twitter access token secret.
  • FACEBOOK_APP_ID: Facebook app ID.
  • FACEBOOK_APP_SECRET: Facebook app secret.
  • FACEBOOK_ACCESS_TOKEN: Facebook access token.
  • FACEBOOK_PAGE_ID: Facebook page ID.
  • INSTAGRAM_ACCESS_TOKEN: Instagram access token.
  • INSTAGRAM_USER_ID: Instagram user ID.

PayPal

  • PAYPAL_CLIENT_ID: PayPal REST client ID.
  • PAYPAL_CLIENT_SECRET: PayPal REST client secret.
  • PAYPAL_API_BASE: Base URL for the PayPal API.

Push Notifications

  • VAPID_PUBLIC_KEY: VAPID public key for web push.
  • VAPID_PRIVATE_KEY: VAPID private key for web push.
  • VAPID_ADMIN_EMAIL: Contact email for VAPID registration.

Trusted Web Activity

  • TWA_SHA256_FINGERPRINT: SHA256 certificate fingerprint for the TWA.

Background Tasks

  • USE_TASK_QUEUE: Enable Redis-backed task queue when true.
  • REDIS_URL: Redis connection URL for the task queue.

Key Components

Flask Blueprints

The application is modularized using Flask Blueprints:

  • main_bp: Main routes and views (defined in main.py).
  • admin_bp: Admin-specific routes and views (defined in admin.py).
  • auth_bp: Authentication routes and views (defined in auth.py).
  • badges_bp: Badge management routes and views (defined in badges.py).
  • ai_bp: AI quest generation routes and views (defined in ai.py).
  • profile_bp: User profile management routes and views (defined in profile.py).
  • quests_bp: Quest management and submission routes and views (defined in quests.py).

Database Models

The database models live in app/models/ and include:

  • User: Represents a user in the system.
  • Game: Represents a game that users can participate in.
  • Quest: Represents a quest that users can complete.
  • Badge: Represents a badge that users can earn.
  • QuestSubmission: Represents a user's submission for a quest.
  • UserQuest: Tracks a user's progress on a quest.
  • ShoutBoardMessage: Represents a message posted on the Shout Board.
  • Sponsor: Details sponsors associated with a game.
  • Notification: Stores notifications sent to users.

Forms

Forms are defined using WTForms in app/forms.py and include:

  • ProfileForm: Form for updating user profiles with a timezone selector. Admin upgrades are handled through a PayPal subscription modal.
  • ShoutBoardForm: Form for posting messages on the Shout Board.
  • QuestForm: Form for creating and updating quests.
  • PhotoForm: Form for submitting photos or videos for quest verification.
  • ContactForm: Form for contacting support.

Utilities

Utility functions are organized under app/utils/ and include:

  • save_profile_picture: Saves profile pictures.
  • save_badge_image: Saves badge images.
  • update_user_score: Updates a user's score.
  • check_and_award_badges: Checks quest completion and awards badges.
  • can_complete_quest: Checks if a user can complete a quest.
  • get_int_param: Safely parses integers from request data.
  • send_email: Sends emails.

Bot Protection

The application uses flask-Humanify with a grid challenge on login and registration endpoints, reducing automated abuse.

Notifications and Push

Email and browser notifications keep users informed:

  • notifications.py: Helper functions for sending emails and Web Push messages.
  • push.py: REST endpoints for managing subscriptions and dispatching push payloads.

Background Tasks

Long running work is executed outside of the request cycle:

  • tasks.py: RQ task definitions for jobs like media processing and email delivery.
  • scheduler.py: Configures recurring jobs using APScheduler.

Admin Functionality

Admin Dashboard

The admin dashboard provides an overview of the platform's activity and allows admins to manage various aspects of the system. It includes:

  • User Management: View and manage user accounts.
  • Game Management: Create, update, and delete games.
  • Quest Management: Create, update, and delete quests.
  • Badge Management: Create, update, and delete badges.
  • Shout Board Management: View and delete Shout Board messages.

Demo Game Rotation

When a new demo game is created automatically, any existing demo games are archived. Archived games remain visible on user profiles along with their points and submissions but are hidden from the join game modal. Users cannot select or join an archived demo game. All photo submissions for archived demo games are removed from storage to conserve space.

Badge Management

Admins can manage badges using the following routes:

  • Create Badge: /badges/create
  • Manage Badges: /badges/manage_badges
  • Update Badge: /badges/update/<int:badge_id>
  • Delete Badge: /badges/delete/<int:badge_id>

Quest Management

Admins can manage quests using the following routes:

  • Create Quest: /quests/<int:game_id>/add_quest
  • Manage Quests: /quests/<int:game_id>/manage_quests
  • Update Quest: /quests/quest/<int:quest_id>/update
  • Delete Quest: /quests/quest/<int:quest_id>/delete

Badge Options

Each quest has a badge_option field determining how badges are awarded:

  • none – the quest only gives points.
  • individual – the quest awards its own badge after the required completions.
  • category – completing all quests in the same category awards the category badge.
  • both – the quest awards its own badge and contributes to the category badge.

Select the appropriate option when creating or updating a quest.

Shout Board

The Shout Board allows admins to post and pin messages that are viewable by all users. Admins can manage Shout Board messages using the following routes:

  • Post Message: /shout-board
  • Pin Message: /pin_message/<int:message_id>
  • Delete Message: Managed via the admin dashboard.

User Functionality

Quest Submission

Users can complete quests and submit verification using the following routes:

  • Submit Quest: /quests/quest/<int:quest_id>/submit
  • View Quest Submissions: /quests/quest/<int:quest_id>/submissions
  • Delete Submission: /quests/quest/delete_submission/<int:submission_id> (POST or DELETE)

View User Submissions

Users can view their submissions using the following routes:

  • View My Submissions: /quests/quest/my_submissions
  • Delete My Submission: /quests/quest/delete_submission/<int:submission_id> (POST or DELETE)

Profile Management

Users can manage their profiles using the following routes:

  • View Profile: /profile/<int:user_id>
  • Edit Profile: /profile/<int:user_id>/edit
  • Upgrading via the edit route sets is_admin to True and applies default game limits (5 GB storage and 60 day retention).
  • Set Timezone: /profile/<int:user_id>/timezone
  • Post Message on Profile Wall: /profile/<int:user_id>/messages
  • Delete Profile Wall Message: /profile/<int:user_id>/messages/<int:message_id>/delete
  • Reply to Profile Wall Message: /profile/<int:user_id>/messages/<int:message_id>/reply
  • Edit Profile Wall Message: /profile/<int:user_id>/messages/<int:message_id>/edit

Social Media Integration

Users can integrate their social media accounts to share their achievements:

  • Twitter Integration: Post quest completions on Twitter.
  • Facebook Integration: Share quest completions and photos on Facebook.
  • Instagram Integration: Post photos related to quest completions on Instagram.

AI Quest Generation

Our platform leverages AI to generate new quests for users. This functionality is powered by OpenAI and involves the following steps:

  1. Generate Quest: /ai/generate_quest

    • This route accepts a quest description and uses OpenAI to generate quest details such as title, description, tips, points, completion limit, frequency, verification type, badge name, and badge description.

  2. Create Quest: /ai/create_quest

    • After generating quest details, this route creates the quest in the database.

  3. Generate Badge Image: /ai/generate_badge_image

    • This route uses OpenAI to generate a badge image based on the badge description.

Static Assets

CSS and JavaScript

Third-party libraries (Bootstrap, KaTeX, Font Awesome, and Highlight.js) are loaded from public CDNs. All custom JavaScript lives in frontend/ and is bundled with Vite into app/static/dist/. SCSS files under app/static/scss/ are also processed by Vite and emitted as style.css in the same directory. After modifying JavaScript or SCSS, run npm install (once) and then npm run build to generate the latest bundles.

Images and Videos

Images and videos are located in app/static/images, app/static/qr_codes, and app/static/videos.

Uploads are validated server-side:

  • Images: PNG, JPG/JPEG, GIF or WebP, maximum 8 MB and 4096×4096 pixels.
  • Videos: MP4, WebM or MOV, maximum 10 MB and 1920×1080 pixels.

Testing and Debugging

Running Tests

Tests are crucial for maintaining the integrity of the codebase. To run tests:

  1. Navigate to the project directory: ```bash cd your-project ```

  2. Run tests: ```bash poetry run pytest

If you have global pytest plugins that interfere, you can isolate the run:

``` PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 poetry run pytest ``` ```

Debugging

To enable debugging, set DEBUG=true in your .env file. This enables Flask's debugger with detailed error messages and an interactive browser debugger.

If login requests fail with a "CSRF session token is missing" message, the session cookie was not sent. Ensure SECRET_KEY is defined and access the site using the host specified by LOCAL_DOMAIN (typically localhost:5000).

JavaScript helpers always include CSRF tokens. To disable CSRF checks in a local environment, set WTF_CSRF_ENABLED=false in your .env file.

JSON endpoints must send the token in the X-CSRF-Token header. The server rejects state-changing JSON requests that omit or provide an incorrect header value.

When DEBUG=true, JavaScript helpers skip sending CSRF tokens to simplify local testing. CSRF protection on the server is also disabled in this mode so requests succeed without the token.

Log messages are saved to logs/application.log. Debug messages only appear when the application runs with DEBUG=true or flask run --debug.

Deployment

Production Configuration

Before deploying to production, ensure the following settings in your .env file:

  • DEBUG = false
  • SESSION_COOKIE_SECURE = true
  • SESSION_COOKIE_DOMAIN should match your production domain or be left blank to use the request host.
  • SQLALCHEMY_DATABASE_URI is set to the production database URL.
  • SECRET_KEY is set to a secure value.

Deployment Steps

  1. Set up the server:

    • Install required packages: Python, PostgreSQL, Nginx, Gunicorn.

    • NGINX Config: ```

    Serve robots.txt

    location = /robots.txt { alias /var/www/html/app/robots.txt; } a

    Security Headers

    add_header Strict-Transport-Security "max-age=3600; includeSubDomains" always; add_header X-Content-Type-Options nosniff; add_header X-Frame-Options DENY; add_header X-XSS-Protection "1; mode=block";

    Content Security Policy

    add_header Content-Security-Policy " default-src 'self'; script-src 'self' 'unsafe-inline' https://code.jquery.com https://cdnjs.cloudflare.com https://stackpath.bootstrapcdn.com https://cdn.jsdelivr.net https://code.jquery.com https://cdnjs.cloudflare.com https://www.googletagmanager.com; style-src 'self' 'unsafe-inline' https://stackpath.bootstrapcdn.com https://cdn.jsdelivr.net https://cdnjs.cloudflare.com; img-src 'self' data:; font-src 'self' data: https://cdn.jsdelivr.net https://cdnjs.cloudflare.com; connect-src 'self' https://www.google-analytics.com https://questbycycle.org wss://questbycycle.org; frame-src 'self'; object-src 'none'; base-uri 'self'; form-action 'self'; ";

    Gzip Compression

    gzip on; gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    Serve Static Files - Bundled Assets

    location /static/dist/ { alias /var/www/html/app/static/dist/; expires 1h; # Cache bundled assets for 1 hour add_header Cache-Control "public, max-age=3600, must-revalidate"; try_files $uri $uri/ =404; }

    Serve Static Files - Video

    location /static/videos/ { alias /var/www/html/app/static/videos/; expires 30d; add_header Cache-Control "public, max-age=, max-age=2592000, must-revalidate"; try_files $uri $uri/ =404; }

    Serve Static Files - Photos

    location /static/photos/ { alias /var/www/html/app/static/photos/; expires 30d; # Cache photos for 30 days add_header Cache-Control "public, max-age=2592000, must-revalidate"; try_files $uri $uri/ =404; }

    Serve Other Static Files

    location /static/ { alias /var/www/html/app/static/; expires 1h; # Cache other static files for 1 hour add_header Cache-Control "public, max-age=3600, must-revalidate"; try_files $uri $uri/ =404; }

    Proxy Pass for Application

    location / { proxy_pass http://127.0.0.1:5000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; }

    client_max_body_size 10M;

    First attempt to serve request as file, then as directory, then fall back to displaying a 404.

    try_files $uri $uri/ =404; } ```

  2. Clone the repository: ```bash git clone https://github.com/your-repo/your-project.git cd your-project ```

  3. Set up the virtual environment: ```bash python3 -m venv venv source venv/bin/activate ```

  4. Install dependencies: ```bash poetry install ```

  5. Configure the database:

    • Update the .env file with the production database credentials.

  6. Run database migrations: ```bash flask db upgrade ```

    Note: As of 2025-08-30, a new column foreign_actor.created_at was added to track when a remote actor cache entry was first created. If your deployment uses Alembic/Flask-Migrate, generate and apply a migration: ```bash flask db migrate -m "Add created_at to foreign_actor" flask db upgrade ```

  7. Set up Gunicorn: ```bash gunicorn --bind 0.0.0.0:8000 wsgi:app ```

  8. Configure Nginx:

    • Set up an Nginx server block to proxy requests to Gunicorn.

  9. Start the application:

    • Ensure Gunicorn and Nginx are running.

  10. Start the task worker:

    • Run poetry run rqworker to process background jobs.

By following this guide, you should have a comprehensive understanding of the project's architecture, codebase, and development practices. Happy coding!