Skip to content

README.md

Latest commit

 

History

History
346 lines (227 loc) · 15.4 KB

README.md

File metadata and controls

346 lines (227 loc) · 15.4 KB
Raw

Book Formatter UI & Local Worker

Full-stack Next.js (App Router) book formatting application: manuscript ingest → metadata & cover configuration → build job with step progression → EPUB and PDF artifact generation. Features real-time SSE events, customizable watermarks, and Kindle compatibility.

Core Features (Implemented)

Ingest & Upload

  • Drag-and-drop multi-file uploads with duplicate detection, type/size validation, partial success (207 Multi-Status), JSONL logging, and per-file error surfacing.
  • Local filesystem persistence (web/data/uploads) + file metadata inference.
  • Upload list persisted client-side (excluding File blobs) with de-duping and drag reordering + removal.
  • Per-file wizard flow: auto-advances when each stage (metadata, cover) is saved; final cover save auto-triggers build job.

Global App State

  • Context-driven state for metadata, cover config, and build options with debounced localStorage persistence.
  • Controlled forms with validation feedback and toast confirmations.
  • Per-file metadata & cover state maps keyed by file ID with route guarding (cannot access Metadata/Cover without at least one file).

Job & Build Pipeline

  • Unified job model (jobType: ingest | build) stored in jobs.json with steps + artifacts.
  • Worker loop auto-started in Node runtime processing one pending job at a time.
  • Build jobs have step timeline (running, done, error, canceled) and emit normalized SSE events.
  • Cancellation mid-step updates job + remaining steps and emits events.
  • Failure injection + accelerated durations via environment flags.
  • Automatic build job creation after final per-file cover save (when user chooses Save & Build / or final wizard step).

Real-Time Events

  • Server-Sent Events (/api/events) broadcasting normalized envelopes: { id, type, timestamp, jobId, jobType, ... } for: started, progress, step, artifacts, completed, canceled, failed.
  • Toast notifications for completion, cancellation, failure, and downloads.

Artifacts

  • EPUB and PDF files generated with Puppeteer-based HTML conversion.
  • Customizable watermarks with position, opacity, color, and size controls.
  • Chapter detection with support for bonus chapters and subtitles.
  • Kindle compatibility with proper EPUB3 metadata and TOC structure.
  • Files become ready at build completion and are downloadable via /api/downloads/:jobId/:type.

Jobs Dashboard

  • Timeline UI for build steps; progress bar for ingest jobs.
  • Filtering (type, status, ID search) + sorting (created, progress, status, type).
  • Export filtered set as JSON or CSV.
  • Accessible controls with persisted filter + sort state.
  • Distinct styling for canceled vs failed vs completed; artifact readiness toasts.

UX & Feedback

  • Global toast system with auto-dismiss and variants.
  • Theme switching (DaisyUI v5) with persisted preference.
  • Responsive layout & accessible labels.

Theming (DaisyUI v5)

This project uses DaisyUI v5 with an explicit plugin invocation and a single centralized theme definition file: web/daisyThemes.js.

Key points:

  • Built-in themes are listed (with optional flags) in BUILT_IN_THEMES.
    • Use --default to mark the initial light theme.
    • Use --prefersdark to bind a theme to the user's OS dark preference.
  • A custom bookish theme is defined via CSS custom properties (v5 variable naming) in BOOKISH_THEME and injected by a small Tailwind plugin in tailwind.config.js.
  • The UI consumes UI_THEMES, which is derived from the build list (strips flags) plus the custom theme (system option intentionally omitted for a fixed explicit list). This guarantees the dropdown always matches what Tailwind compiled.
  • To add or remove a theme: edit BUILT_IN_THEMES, restart the dev server (Tailwind must re-scan), and optionally re-order for display priority.

File overview:

  • web/daisyThemes.js – single source of truth (exports BUILT_IN_THEMES, BOOKISH_THEME, UI_THEMES).
  • web/tailwind.config.js – invokes daisyui({ themes: BUILT_IN_THEMES }) and injects the custom theme with addBase.
  • web/components/TopNav.tsx – imports UI_THEMES for the theme selector.

Why centralization? Avoids mismatch between compiled themes and UI options, simplifies future theme trimming, and cleanly encapsulates the custom palette.

Migration notes (v4 → v5):

  • v5 ignores the legacy daisyui: { themes: [...] } array if flags or plugin options aren’t supplied correctly; we now pass options directly to the plugin call.
  • Theme selection flags replace old implicit behavior; the project uses explicit --default and --prefersdark for deterministic system behavior.
  • tailwind.config.ts now intentionally delegates to tailwind.config.js so the JS file remains the single source of truth for plugin options and themes.

Testing & Simulation

Environment flags (set before npm run dev or in .env.local):

TEST_MODE=1          # Speeds up step durations & ingest progress loop
FAIL_POINTS=BUILD_STEP2,INGEST_50  # Comma list of injected failures

Supported failure tokens:

  • BUILD_STEPN (N = 1-based step index) – marks step as error and job failed.
  • INGEST_P (P = integer progress %) – fails ingest job when progress matches.

Behavior under TEST_MODE:

  • Build step durations shrink (~150–300ms per step).
  • Ingest loop total duration reduced (~600ms base).

Project Structure Highlights

Testing

Integration tests are under web/__tests__/integration. To run them locally:

Step 1. Install dependencies in web:

cd web
npm install

Step 2. Run the integration tests:

npm run test:integration

CI: The workflow .github/workflows/integration-tests.yml installs Playwright browsers (if available) and runs the integration tests in the web package.

For Windows-specific install tips and browser/runtime setup see docs/system-requirements.md.

  • web/lib/jobStore.ts File-backed CRUD for jobs & manuscripts.
  • web/lib/jobProcessor.ts Worker loop + step progression, cancellation checks, failure injection.
  • web/lib/events.ts Server event bus.
  • web/app/api/... Route handlers (uploads, build create, jobs list/cancel, downloads, events, admin stubs).
  • web/app/state/* React context providers (AppState, Toasts).
  • web/types/domain.ts Shared types (Job, JobStep, Artifact, User stubs, etc.).

Running Locally

npm install
npm run dev
# Visit http://localhost:3000

Environment Variables

The application supports the following environment variables (set in .env.local or export before running):

Required for Production

  • JOB_WORKER=1 - Enables the background job processor (required for build jobs to execute)

PDF Generation (Railway/Production)

  • PUPPETEER_EXECUTABLE_PATH - Path to Chrome/Chromium executable (e.g., /usr/bin/chromium)
  • CHROME_PATH - Alternative to PUPPETEER_EXECUTABLE_PATH
  • CHROME_BIN - Alternative to PUPPETEER_EXECUTABLE_PATH

At least one Chrome path variable should be set on Railway. The app will auto-detect system Chrome on macOS/Linux if not set.

Deployment Notice Banner

  • DEPLOYMENT_NOTICE_ENABLED=true - Shows a banner at the top of the app
  • DEPLOYMENT_NOTICE_MESSAGE - Text to display in the banner (e.g., "NOTICE: An update will be deployed...")
  • DEPLOYMENT_NOTICE_LINK - Optional URL for the banner to link to

Testing & Development

  • TEST_MODE=1 - Speeds up step durations and ingest progress loop for testing
  • FAIL_POINTS=BUILD_STEP2,INGEST_50 - Comma-separated list of injected failure points
  • NODE_ENV=test - Automatically enables TEST_MODE

Advanced Configuration

  • STEP_TIMEOUT_MS - Timeout for each build step in milliseconds (default: 30000, TEST_MODE: 1000)
  • STEP_MAX_RETRIES - Number of retries for failed steps (default: 1, TEST_MODE: 0)
  • ARTIFACT_GEN_RETRIES - Number of retries for artifact generation (default: 1, TEST_MODE: 0)
  • PDF_ENGINE=chromium - PDF generation engine (currently only chromium supported)
  • NEXT_DIST_DIR - Custom Next.js build output directory (useful to avoid sync client locks on Windows)

Example .env.local for Production (Railway)

JOB_WORKER=1
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
DEPLOYMENT_NOTICE_ENABLED=false

Example .env.local for Development

JOB_WORKER=1
TEST_MODE=1
NODE_ENV=development

Railway Deployment Configuration

When deploying to Railway, configure the following settings in your service:

Build Settings

  • Custom Build Command: npm run build --workspace=book-formatter-web
  • Watch Paths: /web/** (triggers deployment only when web files change)

Deploy Settings

  • Custom Start Command: npm run start --workspace=book-formatter-web

Apt Packages (System Dependencies)

Required for PDF generation with Puppeteer:

  • chromium - Chrome/Chromium browser for PDF rendering

To add in Railway dashboard:

  1. Go to your service → Settings → Deploy
  2. Find "Apt Packages" section
  3. Add: chromium

Railway Environment Variables

Set these in Railway dashboard under Variables:

  • JOB_WORKER=1 (required)
  • PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium (or leave blank for auto-detection)
  • DEPLOYMENT_NOTICE_ENABLED=false (optional)

Railway automatically provides:

  • RAILWAY_ENVIRONMENT_NAME - Environment name (production, staging, etc.)
  • RAILWAY_PROJECT_ID - Your project ID
  • PORT - Port for the app to listen on (default: 8080)

Notes

  • Railway uses Railpack (v0.13.0) which auto-detects Node.js workspace monorepos
  • The build process automatically runs in the workspace root with proper npm workspace detection
  • PDF generation requires Chromium to be installed via apt packages (not just as a build dependency)

Optional with simulation flags:

TEST_MODE=1 FAIL_POINTS=BUILD_STEP3 npm run dev

On Windows (PowerShell) you can enable the background job worker in two ways:

  • Persist across runs: create web/.env.local with JOB_WORKER=1 (already added in this repo).
  • One-off in PowerShell: run:
$env:JOB_WORKER = '1'; npm run dev

Note: set JOB_WORKER=1 is a cmd.exe style assignment and won't export to PowerShell child processes. Use the PowerShell form above or .\web\.env.local for next dev to pick it up.

Dependencies Report

Generate an up-to-date dependency summary (with resolved versions) into docs/dependencies.md.

npm run docs:deps
  • Output: docs/dependencies.md
  • Lockfile: uses package-lock.json if present to show resolved versions

Current Status

Core Pipeline: ✅ Complete - Full workflow from upload through metadata/cover configuration to artifact generation (EPUB + PDF with watermarks).

Artifact Generation: ✅ Complete - Real EPUB and PDF generation with Puppeteer, customizable watermarks, chapter detection, and Kindle compatibility.

Job Management: ✅ Complete - SSE events, cancellation, filtering/sorting, JSON/CSV export, timeline UI.

Pending Work: Auth integration, retention/cleanup policies, batch retry operations, progress smoothing.

Planned / Remaining Work

  • Retry / restart flows (single + batch, lineage via parentJobId) for failed/canceled jobs.
  • Retention: prune old jobs, uploads, logs (configurable TTL + sweep utility).
  • Auth integration + per-user scoped events & entitlement gating.
  • Step-level heuristics (ETA + progress smoothing) and richer diagnostics panel.
  • Artifact metadata enrichment (checksum, size, content-type for generated files).
  • Structured JSONL event logging with correlation IDs.
  • Accessibility: keyboard-based upload reordering, high-contrast theme audit.

Auth & Entitlement Stubs (Inactive)

See web/lib/auth.ts, web/lib/userStore.ts, web/lib/entitlements.ts and admin API stubs. Enable after choosing provider:

  1. Implement getSession.
  2. Provision user + initial entitlement sync.
  3. Gate admin + worker endpoints.
  4. Filter SSE events by userId.

Accessibility & Conventions

  • All interactive controls have labels or aria-label.
  • Color/status indicators use icon + text.
  • LocalStorage keys namespaced and debounced to reduce churn.

Export Data

From Jobs page: export filtered set as JSON or CSV for manual inspection or test artifacts.

Contributing / Notes

  • Keep new events normalized (extend SSE normalization switch only).
  • Avoid introducing new step statuses without updating UI & docs.
  • Prefer additive file-based persistence; migrations unnecessary during prototype.

Troubleshooting

Symptom Cause Fix
ENOTEMPTY / EPERM during npm install on Windows Sync clients (OneDrive/Dropbox) or open editors locking files inside node_modules or temp build folders Pause or disable sync clients, close editors/terminals that may hold handles, remove node_modules (root and web/node_modules), run npm cache clean --force, then cd web && npm install --legacy-peer-deps. See docs/system-requirements.md for PowerShell commands.
UNKNOWN error opening server-reference-manifest.json or react-loadable-manifest.json (Windows + Dropbox) Sync client locking build artifacts Use a non-synced build dir. EITHER keep web/next.config.js with distDir: 'node_modules/.next' OR keep web/next.config.ts and set NEXT_DIST_DIR to a non-synced path. Ensure only one config file remains.
EPERM rename errors when starting dev on Windows with Dropbox Dropbox or other sync service locking webpack cache files in .next Run the included cache-clear script: pwsh -File web\scripts\clear-next-cache.ps1 and restart dev (npm run dev). The project also places webpack's dev cache in your OS temp dir to avoid future locks (see web/next.config.js).
Stale job list after restart Worker not yet picked next job Wait for interval or trigger a new upload/build
SSE disconnects on long idle Browser/network timeout Page auto-reconnects; refresh if no events after several minutes
Artifact buttons disabled Artifacts not yet marked ready Wait for build completion; ensure no injected failure

Optional: set a custom build output directory (outside synced folders) by exporting NEXT_DIST_DIR before starting dev.

set NEXT_DIST_DIR=.next-cache && npm run dev   # Windows PowerShell (example)

Changelog

See CHANGELOG.md for detailed version history.

Latest Updates (2025-11-21)

PDF Generation Overhaul - Replaced PDFKit with Puppeteer-core for HTML-to-PDF conversion with full watermark support (customizable text, position, opacity, color, size). Includes PDF validation with magic byte checking to prevent corrupted files. Optimized rendering from 160 DOM watermark elements to 15-element CSS grid for faster processing.

Performance Optimization - Increased PDF generation timeout from 30s to 60s for large documents. Changed page load strategy from 'networkidle0' to 'domcontentloaded' for faster rendering without waiting for all network requests.

EPUB Improvements - Removed cover page from table of contents to match standard EPUB conventions. Improved chapter detection for bonus chapters with subtitles. Enhanced watermark compatibility for Kindle readers.

Documentation - Added comprehensive environment variables guide, Railway deployment configuration with Chromium apt package requirements, and complete changelog of all 171 commits since project inception.

Accessibility & Fixes - Added aria-labels to all watermark form controls. Fixed duplicate chapter headings in PDF output. Fixed TypeScript errors in Next.js 15 DELETE routes.

License

Prototype / internal evaluation. Licensing TBD.