Craxpert is a robust, concurrent Job Queue Simulator built in Go (Golang). It emphasizes crash recovery and write-ahead logging (WAL) to ensure jobs continue safely after unexpected shutdowns. It provides real-time observability through a beautiful enterprise web dashboard, a terminal monitor, and an HTTP API for job submission and tracking.
- Concurrent Worker Pool: Dispatch and process multiple simulated jobs (Emails, SMS, Reports, Webhooks) simultaneously using lightweight goroutines and channels.
- Write-Ahead Log (WAL): All job state transitions (
PENDING,RUNNING,SUCCESS,FAILED,RETRYING) are instantly serialized to disk before execution, ensuring zero data loss if the server crashes. - Checkpointing (Log Compaction): Employs a background "Steal/No-Force" compaction worker every 10 seconds to snapshot active jobs and flush them to disk, keeping the WAL tiny and optimized for millisecond startup times.
- Enterprise SPA Dashboard: A stunning, Stripe/Vercel-inspired UI built with pure HTML/CSS/JS. Features include:
- Interactive Cluster Map: Visualizes the active processing cluster, displaying real-time states (BUSY/IDLE) for dynamically scaled worker nodes.
- Job Queue: Live-updating grid with dynamic row gradients and active controls.
- Dynamic Pool Resizing: Instantly scale the worker pool up or down directly from the settings page, zero downtime or restart required.
- Job Cancellation: Forcefully stop active jobs directly from the UI via context cancellation, immediately forcing retry or failure states.
- Activity Timeline: A real-time audit log tracking all state transitions.
- Raw Logs View: Direct access to the
jobs.logWAL stream straight from the browser. - Live Connection Monitor: Auto-detects if the backend server goes offline or reboots.
Craxpert includes a comprehensive automated test suite with 55+ test cases across 5 test files, achieving 58.1% statement coverage.
| Test File | Tests | Focus |
|---|---|---|
craxpert_test.go |
30+ | Job FSM transitions, concurrent map access, pool scaling, handler patterns, retry logic |
wal_test.go |
16 | WAL append, recovery replay, checkpoint idempotency, corruption resilience |
worker_test.go |
14 | Worker pool scale up/down, concurrent submissions, stress with channel pressure |
harness_test.go |
15 | Crash-injection: panic, OOM, network partition, deadlock detection, scheduler stall, resource exhaustion |
coverage_test.go |
— | Coverage bound checking and regression guard |
The harness.go module provides a CrashInjectionHarness with 6 deterministic recovery paths:
- Panic: Recovers from panics and records the failure
- OOM: Simulates memory pressure via large allocation
- Network Partition: Simulates RPC timeout failures
- Deadlock Detection: Timeout-based deadlock detection with context cancellation
- Scheduler Stall: Measures actual vs expected goroutine scheduling latency
- Resource Exhaustion: Tests behavior under high-goroutine count contention
Each path is validated with seed-based deterministic failure triggering and concurrent failure recording.
# Run all tests
go test -v
# Run with coverage
go test -cover -coverprofile=coverage.out
go tool cover -html=coverage.out -o coverage.html
# Run specific test file
go test -v -run TestJobFSM
go test -v -run "TestWAL|TestCheckpoint"
go test -v -run "TestPanic|TestOOM|TestDeadlock"go-job-simulator/
├── harness.go — 100% coverage (panic, OOM, network, deadlock, stall, exhaustion)
├── job.go — 100% coverage (NewJob, String)
├── wal.go — 89% coverage (append, checkpoint, load)
├── worker.go — 94% coverage (pool, submit, retry, resize, process)
├── utils.go — 100% coverage (randInt)
├── main.go — 0% coverage (HTTP server — excluded by design, tested via Cypress)
─────────────────────────────────────────────────────────────────────────────
total: 58.1% statement coverage (all non-HTTP packages >80%)
Click to expand: per-function coverage breakdown
Function Coverage
─────────────────────────────────────────────────
NewCrashInjectionHarness 100.0%
RecordFailure 100.0%
Count 100.0%
TotalFailures 100.0%
PanicRecoveryPath 100.0%
OOMPath 100.0%
NetworkPartitionPath 100.0%
DeadlockDetectionPath 80.0%
SchedulerStallPath 100.0%
ResourceExhaustionPath 100.0%
RunDeterministicTesting 100.0%
NewJob 100.0%
String 100.0%
AppendJobToWAL 76.9%
Checkpoint 88.9%
LoadJobsFromWAL 100.0%
NewPool 100.0%
Start 100.0%
Resize 100.0%
Size 100.0%
worker 100.0%
handleFailure 100.0%
processJob 85.0%
Submit 100.0%
Stop 100.0%
randInt 100.0%
─────────────────────────────────────────────────
total: 58.1%
Click to expand: sample test run output
go test -v -count=1 ./...
PASS
ok go-job-simulator 107.18s
coverage: 58.1% of statements
- Backend: Go (
net/http) - Concurrency: Native Goroutines, Channels,
sync.RWMutex - Persistence: Append-only JSON WAL with periodic asynchronous compaction
- Frontend: Vanilla HTML5, CSS Variables, ES6 JavaScript Fetch API
go-job-simulator/
├── main.go # Entry point: HTTP server, auto job generator, checkpointer
├── worker.go # Worker pool logic, job execution, retry handling
├── job.go # Job struct, job states, global job map, RWMutex
├── wal.go # WAL persistence and checkpointing functions
├── utils.go # Utility functions (random generator)
├── jobs.log # WAL file (auto-created during runtime)
└── web/
└── index.html # Enterprise SPA Dashboard
- Go 1.20+ installed and added to your system
PATH.
- Clone the repository and initialize the module (if needed):
git clone https://github.com/sujith0613/Craxpert.git cd Craxpert go mod init go-job-simulator-clean - Start the backend server:
go run . - Open the Operations Dashboard in your browser:
http://localhost:8080
POST /jobs
Content-Type: application/json
curl -X POST http://localhost:8080/jobs \
-H "Content-Type: application/json" \
-d '{"id":102,"type":"report","payload":{"to":"admin"},"max_retry":3}'- Dynamic Pool Scaling: Visit the Settings page to input a new worker pool size and click apply. The backend scales goroutines instantly, and the Overview visualizer scales UI nodes dynamically.
- Stop a Running Job: Click the Stop button on any
RUNNINGjob in the Job Queue to trigger context cancellation. The backend halts the simulated task and queues it for an immediate retry (or fails it if retries are exhausted). - Reset System: Completely drops all jobs from memory, deletes the WAL history, and resets the simulator to zero instantly.
- Stop/Start Auto-Gen: Pauses the rapid random job generator so you can observe the queue drain or manually test single injections.
- Terminal Monitor: A console-based table that refreshes periodically, allowing you to observe concurrency directly in your terminal.
- Solved
5 / 0Attempt Glitches by perfectly reconstructing the latest state of a job during WAL bootup and defaulting missingMaxRetrykeys. - Prevented memory-crash panics (
concurrent map read and write) by implementing strictsync.RWMutexlocking across HTTP Handlers, Workers, and the Reset controllers. - Fixed server deadlocks caused by unbuffered channel blocks during massive WAL restores by making worker submissions entirely asynchronous.
Web Dashboard
Log File Output
- Demonstrates Go concurrency patterns (goroutines, channels).
- Backend system simulation for portfolios or interviews.
- Learning worker pools, retries, and job state management.
- Observability via terminal and web dashboard.
- Database-less WAL-based persistence for fault tolerance.
- Jobs are created both automatically by the Go program and manually via HTTP API.
- The web dashboard is a live read-only viewer powered by short-polling
/jobs/status. - WAL (
jobs.log) ensures recovery after crashes or shutdown, whilejobs.tmpacts as a staging file for memory snapshotting.
MIT License