Create streamlined 'golden path' installer for new users

.env.example

@@ -0,0 +1,45 @@
+# Spyglass Docker Compose Configuration
+#
+# This file shows available configuration options with their defaults.
+# Docker Compose works fine without a .env file - defaults work for most users.
+#
+# To customize (optional):
+#   1. Copy this file: cp .env.example .env
+#   2. Edit .env with your preferred settings
+#   3. Run: docker compose up -d
+#
+# IMPORTANT: If you change MYSQL_PORT or MYSQL_ROOT_PASSWORD, you must also
+# update your DataJoint configuration file (dj_local_conf.json) to match.
+# See docs/DATABASE.md for details on DataJoint configuration.
+
+# =============================================================================
+# Database Configuration (Required)
+# =============================================================================
+
+# MySQL root password (default: tutorial)
+# For local development, 'tutorial' is fine
+# For production, use a strong password
+MYSQL_ROOT_PASSWORD=tutorial
+
+# MySQL port (default: 3306)
+# Change this if port 3306 is already in use
+MYSQL_PORT=3306
+
+# MySQL Docker image (default: datajoint/mysql:8.0)
+# You can specify a different version if needed
+MYSQL_IMAGE=datajoint/mysql:8.0
+
+# =============================================================================
+# Optional Configuration
+# =============================================================================
+
+# Database name to create on startup (optional)
+# Leave empty to skip database creation
+MYSQL_DATABASE=
+
+# =============================================================================
+# Notes
+# =============================================================================
+# - Don't commit .env file to git (it contains passwords)
+# - Port range: 1024-65535
+# - If you change MYSQL_PORT, update your DataJoint config accordingly

.gitignore

@@ -115,6 +115,9 @@ ENV/
 env.bak/
 venv.bak/
 
+# Docker Compose
+docker-compose.override.yml
+
 # Spyder project settings
 .spyderproject
 .spyproject
@@ -165,6 +168,7 @@ temp_nwb/*s
 *.pem
 dj_local_conf*
 !dj_local_conf_example.json
+!/config_schema.json
 
 !/.vscode/extensions.json
 !/.vscode/settings.json

QUICKSTART.md

@@ -0,0 +1,140 @@
+# Spyglass Quickstart
+
+Get from zero to analyzing neural data with Spyglass in just a few commands.
+
+## Choose Your Path
+
+### 👥 Joining an Existing Lab? (Recommended)
+
+If you received database credentials from your lab admin, this is you!
+The installer will:
+
+- Set up your development environment
+- Connect you to your lab's existing database
+- Prompt you to change your temporary password
+- Configure all necessary directories
+
+**Time**: ~5 minutes | **Database**: Connect to lab's existing database
+
+### 🧪 Trying Spyglass Locally?
+
+Want to explore Spyglass features without joining a lab?
+The installer can:
+
+- Set up a local trial database using Docker
+- Create an isolated test environment
+- Let you experiment with sample data
+
+**Time**: ~10 minutes | **Database**: Local Docker container
+(requires [Docker Desktop](https://docs.docker.com/get-docker/))
+
+---
+
+## Prerequisites
+
+- **Python**: Version 3.9 or higher
+- **Disk Space**: ~10GB for installation + data storage
+- **Operating System**: macOS or Linux (Windows experimental)
+- **Package Manager**: [conda](https://docs.conda.io/) (23.10.0+ recommended) or [mamba](https://mamba.readthedocs.io/)
+
+If you don't have mamba/conda, install [miniforge](https://github.com/conda-forge/miniforge#install) first.
+
+## Installation (2 steps)
+
+### Step 1: Run the installer
+
+```bash
+# Clone the repository
+git clone https://github.com/LorenFrankLab/spyglass.git
+cd spyglass
+
+# Run interactive installer
+python scripts/install.py
+```
+
+The installer will prompt you to choose:
+
+1. **Installation type**: Minimal (recommended) or Full
+2. **Database setup**:
+   - **Remote** (recommended for lab members) - Connect to lab's existing database
+   - **Docker** - Local trial database for testing
+   - **Skip** - Configure manually later
+
+If joining a lab, you'll be prompted to change your password during installation.
+
+### Step 2: Validate installation
+
+```bash
+# Activate the environment
+conda activate spyglass
+
+# Run validation
+python scripts/validate.py -v
+```
+
+**That's it!** Setup complete in ~5-10 minutes.
+
+## Next Steps
+
+### Run first tutorial
+```bash
+cd notebooks
+jupyter notebook 01_Concepts.ipynb
+```
+
+### Configure for your data
+- Place NWB files in `~/spyglass_data/raw/`
+- See [Data Import Guide](https://lorenfranklab.github.io/spyglass/latest/notebooks/01_Insert_Data/) for details
+
+### Join community
+- 📖 [Documentation](https://lorenfranklab.github.io/spyglass/)
+- 💬 [GitHub Discussions](https://github.com/LorenFrankLab/spyglass/discussions)
+- 🐛 [Report Issues](https://github.com/LorenFrankLab/spyglass/issues)
+
+---
+
+## Installation Options
+
+Need something different? The installer supports these options:
+
+```bash
+python scripts/install.py --full           # All optional dependencies
+python scripts/install.py --pipeline=dlc   # DeepLabCut pipeline
+python scripts/install.py --no-database    # Skip database setup
+python scripts/install.py --help           # See all options
+```
+
+## What Gets Installed
+
+The installer creates:
+
+- **Conda environment** with Spyglass and core dependencies
+- **Database connection** (remote lab database OR local Docker container)
+- **Data directories** in `~/spyglass_data/`
+- **Jupyter environment** for running tutorials
+
+## Troubleshooting
+
+### Installation fails?
+```bash
+# Remove environment and retry
+conda env remove -n spyglass
+python scripts/install.py
+```
+
+### Validation fails?
+
+1. Check error messages for specific issues
+2. If using Docker database, ensure Docker Desktop is running
+3. If database connection fails, verify credentials with your lab admin
+4. Try skipping database: `python scripts/install.py --no-database`
+
+### Need help?
+
+- Check [Advanced Setup Guide](https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/) for manual installation
+- Ask questions in [GitHub Discussions](https://github.com/LorenFrankLab/spyglass/discussions)
+
+---
+
+**Next tutorial**: [01_Concepts.ipynb](notebooks/01_Concepts.ipynb)
+**Full documentation**: [lorenfranklab.github.io/spyglass](https://lorenfranklab.github.io/spyglass/)

README.md

@@ -65,10 +65,67 @@ Documentation can be found at -
 
 ## Installation
 
-For installation instructions see -
-[https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/](https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/)
+### Quick Start (Recommended)
 
-Typical installation time is: 5-10 minutes
+Get started with Spyglass in 5 minutes using our automated installer:
+
+```bash
+# Clone the repository
+git clone https://github.com/LorenFrankLab/spyglass.git
+cd spyglass
+
+# Run automated installer
+python scripts/install.py
+
+# Activate environment
+conda activate spyglass
+```
+
+The installer will:
+- ✅ Create conda environment with all dependencies
+- ✅ Set up local MySQL database (Docker) or connect to remote
+- ✅ Validate installation
+- ✅ Provide clear next steps
+
+**Installation Options:**
+```bash
+# Minimal installation (recommended for new users)
+python scripts/install.py --minimal
+
+# Full installation (all features)
+python scripts/install.py --full
+
+# With Docker database
+python scripts/install.py --docker
+
+# Connect to remote database
+python scripts/install.py --remote
+
+# Non-interactive with environment variables
+export SPYGLASS_BASE_DIR=/path/to/data
+python scripts/install.py --minimal --docker
+
+# Non-interactive remote database setup
+python scripts/install.py --remote \
+  --db-host db.lab.edu \
+  --db-user myuser \
+  --db-password mysecret
+
+# Or use environment variable for password
+export SPYGLASS_DB_PASSWORD=mysecret
+python scripts/install.py --remote --db-host db.lab.edu --db-user myuser
+```
+
+**Troubleshooting:**
+- See [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) for common issues
+- Run `python scripts/validate.py` to check your installation
+- For database help, see [docs/DATABASE.md](docs/DATABASE.md)
+
+### Manual Installation
+
+For manual installation and advanced configuration:
+- [Setup Documentation](https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/)
+- [Database Setup Guide](docs/DATABASE.md)
 
 ## Tutorials
 

config_schema.json

@@ -0,0 +1,46 @@
+{
+  "_schema_version": "1.0.0",
+  "_comment": "Single source of truth for Spyglass directory structure",
+  "_critical": "This must match src/spyglass/settings.py SpyglassConfig.relative_dirs",
+  "_note": "If you modify this file, update settings.py to match (or vice versa)",
+  "_version_history": {
+    "1.0.0": "Initial DRY architecture - JSON schema replaces hard-coded directory structure"
+  },
+
+  "directory_schema": {
+    "spyglass": {
+      "raw": "raw",
+      "analysis": "analysis",
+      "recording": "recording",
+      "sorting": "spikesorting",
+      "waveforms": "waveforms",
+      "temp": "tmp",
+      "video": "video",
+      "export": "export"
+    },
+    "kachery": {
+      "cloud": ".kachery-cloud",
+      "storage": "kachery_storage",
+      "temp": "tmp"
+    },
+    "dlc": {
+      "project": "projects",
+      "video": "video",
+      "output": "output"
+    },
+    "moseq": {
+      "project": "projects",
+      "video": "video"
+    }
+  },
+
+  "tls": {
+    "_description": "TLS (Transport Layer Security) encrypts database traffic",
+    "auto_enable_for_remote": true,
+    "localhost_addresses": [
+      "localhost",
+      "127.0.0.1",
+      "::1"
+    ]
+  }
+}

docker-compose.yml

@@ -0,0 +1,68 @@
+# Spyglass Database Setup with Docker Compose
+#
+# Quick start (no setup needed - defaults work for most users):
+#   docker compose up -d
+#
+# This starts a MySQL database for Spyglass with:
+#   - Persistent data storage (survives container restart)
+#   - Health checks (ensures database is ready)
+#   - Standard configuration (matches manual Docker setup)
+#
+# Common tasks:
+#   Start:    docker compose up -d
+#   Stop:     docker compose stop
+#   Logs:     docker compose logs mysql
+#   Restart:  docker compose restart
+#   Remove:   docker compose down -v  # WARNING: Deletes all data!
+#
+# Customization (optional):
+#   Create .env file from .env.example to customize settings
+#   See .env.example for available configuration options
+#
+# Troubleshooting:
+#   Port 3306 in use: Create .env file and change MYSQL_PORT
+#   Services won't start: Run 'docker compose logs' to see errors
+#   Can't connect: Ensure Docker Desktop is running
+
+services:
+  mysql:
+    image: ${MYSQL_IMAGE:-datajoint/mysql:8.0}
+
+    # Container name MUST be 'spyglass-db' to match existing code
+    container_name: spyglass-db
+
+    ports:
+      - "${MYSQL_PORT:-3306}:3306"
+
+    environment:
+      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD:-tutorial}
+      # Optional: Create database on startup
+      MYSQL_DATABASE: ${MYSQL_DATABASE:-}
+
+    volumes:
+      # Named volume for persistent storage
+      # Data survives 'docker compose down' but is removed by 'down -v'
+      - spyglass-db-data:/var/lib/mysql
+
+    healthcheck:
+      # Check if MySQL is ready without exposing password in process list
+      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "--silent"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+
+    restart: unless-stopped
+
+    networks:
+      - spyglass-network
+
+volumes:
+  spyglass-db-data:
+    # Explicit name for predictability
+    name: spyglass-db-data
+
+networks:
+  spyglass-network:
+    name: spyglass-network
+    driver: bridge