|
| 1 | +# Neurostore: Meta-Analysis Backend Services & Frontend |
| 2 | + |
| 3 | +Neurostore is a containerized neuroimaging meta-analysis platform consisting of two Python Flask backend services (Neurostore and Neurosynth-Compose) and a React/TypeScript frontend. All services are deployed using Docker Compose. |
| 4 | + |
| 5 | +**Always reference these instructions first and fallback to `.github/workflows/workflow.yml` for testing setup help, and then fallback to search or bash commands only when you encounter unexpected information that does not match the info here.** |
| 6 | + |
| 7 | +## Working Effectively |
| 8 | + |
| 9 | +### Initial Setup - CRITICAL: Git Submodules Required |
| 10 | +Initialize Git submodules before any build or development work: |
| 11 | +- `git submodule update --init --recursive` -- takes 1-2 seconds. **NEVER CANCEL**. |
| 12 | +- This downloads OpenAPI specifications and TypeScript SDKs required for builds. |
| 13 | + |
| 14 | +### Build Services - NEVER CANCEL These Commands |
| 15 | +Build times vary significantly. Set appropriate timeouts and **DO NOT CANCEL** builds: |
| 16 | + |
| 17 | +#### Store Service (Neurostore Backend) |
| 18 | +- `cd store && cp .env.example .env` |
| 19 | +- `docker network create nginx-proxy` -- creates shared network |
| 20 | +- `docker compose build` -- takes 1-2 minutes. **NEVER CANCEL. Set timeout to 180+ seconds.** |
| 21 | +- `docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d` -- takes 10-15 seconds |
| 22 | + |
| 23 | +#### Compose Service (Neurosynth-Compose Backend + Frontend) |
| 24 | +- `cd compose && cp .env.example .env` |
| 25 | +- `docker compose build` -- takes 2-3 minutes. **NEVER CANCEL. Set timeout to 240+ seconds.** |
| 26 | +- `docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d` -- takes 5-10 seconds |
| 27 | + |
| 28 | +### Database Setup |
| 29 | +Create and migrate databases for both services: |
| 30 | + |
| 31 | +#### Store Database |
| 32 | +- `docker compose exec -T store-pgsql17 psql -U postgres -c "create database test_db"` |
| 33 | +- `docker compose exec -T store-pgsql17 psql -U postgres -d test_db -c "CREATE EXTENSION IF NOT EXISTS vector;"` |
| 34 | +- `docker compose exec -T neurostore bash -c "flask db merge heads && flask db stamp head && flask db migrate && flask db upgrade"` -- takes 5-10 seconds |
| 35 | +- `docker compose exec -T neurostore bash -c "flask ingest-neurosynth --max-rows 100"` -- takes 5-10 seconds |
| 36 | + |
| 37 | +#### Compose Database |
| 38 | +- `docker compose exec -T compose_pgsql17 psql -U postgres -c "create database test_db"` |
| 39 | +- `docker compose exec -T compose bash -c "flask db merge heads && flask db stamp head && flask db migrate && flask db upgrade"` -- takes 5-10 seconds |
| 40 | + |
| 41 | +### Frontend Development |
| 42 | +Navigate to `compose/neurosynth-frontend/`: |
| 43 | + |
| 44 | +#### Build & Development |
| 45 | +- `cp .env.example .env.dev` |
| 46 | +- `npm install` -- takes 25-30 seconds |
| 47 | +- `npm run build:dev` -- takes 25-35 seconds |
| 48 | +- `npm run start:dev` -- starts development server on http://localhost:3000 |
| 49 | + |
| 50 | +#### Configuration Files Needed |
| 51 | +- `.env.dev` -- development environment configuration |
| 52 | +- `.env.staging` -- staging environment configuration |
| 53 | + |
| 54 | +## Testing - Set Long Timeouts |
| 55 | + |
| 56 | +### Backend Tests - NEVER CANCEL |
| 57 | +**CRITICAL**: Backend tests take significant time. Set appropriate timeouts. |
| 58 | + |
| 59 | +#### Store Backend Tests |
| 60 | +- `docker compose run -e "APP_SETTINGS=neurostore.config.DockerTestConfig" --rm neurostore bash -c "python -m pytest neurostore/tests"` |
| 61 | +- Takes 2-3 minutes. **NEVER CANCEL. Set timeout to 300+ seconds.** |
| 62 | +- Expected: ~229 passed, 12 skipped |
| 63 | + |
| 64 | +#### Compose Backend Tests |
| 65 | +- `docker compose run -e "APP_SETTINGS=neurosynth_compose.config.DockerTestConfig" --rm compose bash -c "python -m pytest neurosynth_compose/tests"` |
| 66 | +- Takes 2-4 minutes. **NEVER CANCEL. Set timeout to 360+ seconds.** |
| 67 | + |
| 68 | +#### Frontend Tests |
| 69 | +- `npm run test` -- takes 50-70 seconds |
| 70 | +- Expected: ~345 tests passing |
| 71 | +- `npm run cy:e2e-headless` -- Cypress E2E tests (requires both backend services running) |
| 72 | + |
| 73 | +## Validation Scenarios |
| 74 | + |
| 75 | +### Always Test These Workflows After Changes |
| 76 | +1. **API Functionality**: `curl http://localhost/api/studies?limit=5` -- Store API should return JSON |
| 77 | +2. **Frontend Build**: Ensure `npm run build:dev` completes without errors |
| 78 | +3. **Backend Migration**: Test database migration commands work |
| 79 | +4. **Docker Services**: All containers should start and stay healthy |
| 80 | + |
| 81 | +### Service URLs |
| 82 | +- **Store API**: http://localhost/api (port 80) |
| 83 | +- **Compose API**: http://localhost:81/api (port 81) |
| 84 | +- **Frontend Dev**: http://localhost:3000 (when running `npm run start:dev`) |
| 85 | +- **pgHero**: http://localhost/pghero (database monitoring) |
| 86 | +- **Grafana**: Available via docker compose (database metrics) |
| 87 | + |
| 88 | +## Linting & Code Quality |
| 89 | + |
| 90 | +### Backend Linting |
| 91 | +- `pip install flake8` (if not installed) |
| 92 | +- `cd store/backend && flake8 ./neurostore` -- takes <1 second |
| 93 | +- `cd compose/backend && flake8 ./neurosynth_compose` -- takes <1 second |
| 94 | + |
| 95 | +### Frontend Linting |
| 96 | +- `npm run lint` -- takes 30-40 seconds |
| 97 | +- **Expected**: ~899 linting issues exist (mostly formatting). This is normal. |
| 98 | +- `npm run lint --fix` -- auto-fixes many issues |
| 99 | + |
| 100 | +## Configuration Notes |
| 101 | + |
| 102 | +### Environment Variables |
| 103 | +Both services use similar `.env` configurations: |
| 104 | +- `POSTGRES_HOST` -- Database host (store-pgsql17 or compose_pgsql17) |
| 105 | +- `POSTGRES_PASSWORD` -- Database password (usually "example") |
| 106 | +- `AUTH0_CLIENT_ID` -- Auth0 integration (can be placeholder for dev) |
| 107 | +- `DEBUG=True` -- Enable debug mode |
| 108 | + |
| 109 | +### Docker Configuration Issues |
| 110 | +Development config uses `test_db` database by default, not the service-named databases. This is intentional for the `DevelopmentConfig` class. |
| 111 | + |
| 112 | +## Directory Structure Reference |
| 113 | +``` |
| 114 | +/ |
| 115 | +├── store/ # Neurostore backend service |
| 116 | +│ ├── backend/ # Python Flask application |
| 117 | +│ │ ├── neurostore/ # Main package |
| 118 | +│ │ ├── migrations/ # Database migrations |
| 119 | +│ │ └── pyproject.toml # Python dependencies |
| 120 | +│ ├── docker-compose.yml # Production config |
| 121 | +│ └── docker-compose.dev.yml # Development overrides |
| 122 | +├── compose/ # Neurosynth-Compose service |
| 123 | +│ ├── backend/ # Python Flask application |
| 124 | +│ │ ├── neurosynth_compose/ # Main package |
| 125 | +│ │ └── pyproject.toml # Python dependencies |
| 126 | +│ ├── neurosynth-frontend/ # React/TypeScript app |
| 127 | +│ │ ├── src/ # Source code |
| 128 | +│ │ ├── package.json # Node.js dependencies |
| 129 | +│ │ └── vite.config.ts # Build configuration |
| 130 | +│ ├── docker-compose.yml # Production config |
| 131 | +│ └── docker-compose.dev.yml # Development overrides |
| 132 | +``` |
| 133 | + |
| 134 | +## Common Issues & Solutions |
| 135 | + |
| 136 | +### Missing OpenAPI Specs Error |
| 137 | +- **Problem**: `FileNotFoundError: neurostore-openapi.yml` or `neurosynth-compose-openapi.yml` |
| 138 | +- **Solution**: Run `git submodule update --init --recursive` |
| 139 | + |
| 140 | +### Missing TypeScript SDK Error |
| 141 | +- **Problem**: `Could not resolve "../neurostore-typescript-sdk"` |
| 142 | +- **Solution**: Initialize git submodules first |
| 143 | + |
| 144 | +### Container Won't Start |
| 145 | +- **Problem**: `Worker failed to boot` or similar gunicorn errors |
| 146 | +- **Solution**: Check OpenAPI submodules are initialized and container logs with `docker compose logs [service]` |
| 147 | + |
| 148 | +### Database Connection Errors |
| 149 | +- **Problem**: `database "test_db" does not exist` |
| 150 | +- **Solution**: Create test databases as shown in setup steps |
| 151 | + |
| 152 | +### Frontend Build Sentry Warnings |
| 153 | +- **Problem**: Sentry API token errors during build |
| 154 | +- **Solution**: Expected in development. Build still succeeds - ignore Sentry warnings. |
| 155 | + |
| 156 | +## Performance Expectations |
| 157 | + |
| 158 | +| Operation | Time | Timeout | |
| 159 | +|-----------|------|---------| |
| 160 | +| Git submodules | 1-2s | 30s | |
| 161 | +| Store build | 1m36s | 180s | |
| 162 | +| Compose build | 1m56s | 240s | |
| 163 | +| Frontend npm install | 26s | 60s | |
| 164 | +| Frontend build | 29s | 90s | |
| 165 | +| Backend tests | 2m30s | 300s | |
| 166 | +| Frontend tests | 59s | 120s | |
| 167 | +| Database migrations | 5s | 30s | |
| 168 | +| Backend linting | <1s | 10s | |
| 169 | +| Frontend linting | 37s | 60s | |
| 170 | + |
| 171 | +**Remember**: Always use `--timeout` parameters longer than these estimates. Builds may take longer on slower systems. |
0 commit comments