Skip to content

Commit c0becb7

Browse files
nicoaleenicoalee
committed
chore: resolve merge conflict
1 parent 937b621 commit c0becb7

File tree

190 files changed

+12848
-6623
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

190 files changed

+12848
-6623
lines changed

‎.codespellrc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22
skip = .git,*.pdf,*.svg,*.ts,*.tsx,package-lock.json,data-*vocabulary.txt,sample_*.csv,CHANGELOG.rst,ingest_neurovault.yml,./compose/neurosynth-frontend/cypress/fixtures
33
# regexes
44
ignore-regex = r"\(\?i\).*\\\\1
5-
ignore-words-list = te,fwe,connexion,zoon,covert,rime
5+
ignore-words-list = te,fwe,connexion,zoon,covert,rime,selectin

‎.github/copilot-instructions.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Neurostore: Meta-Analysis Backend Services & Frontend
2+
3+
Neurostore is a containerized neuroimaging meta-analysis platform consisting of two Python Flask backend services (Neurostore and Neurosynth-Compose) and a React/TypeScript frontend. All services are deployed using Docker Compose.
4+
5+
**Always reference these instructions first and fallback to `.github/workflows/workflow.yml` for testing setup help, and then fallback to search or bash commands only when you encounter unexpected information that does not match the info here.**
6+
7+
## Working Effectively
8+
9+
### Initial Setup - CRITICAL: Git Submodules Required
10+
Initialize Git submodules before any build or development work:
11+
- `git submodule update --init --recursive` -- takes 1-2 seconds. **NEVER CANCEL**.
12+
- This downloads OpenAPI specifications and TypeScript SDKs required for builds.
13+
14+
### Build Services - NEVER CANCEL These Commands
15+
Build times vary significantly. Set appropriate timeouts and **DO NOT CANCEL** builds:
16+
17+
#### Store Service (Neurostore Backend)
18+
- `cd store && cp .env.example .env`
19+
- `docker network create nginx-proxy` -- creates shared network
20+
- `docker compose build` -- takes 1-2 minutes. **NEVER CANCEL. Set timeout to 180+ seconds.**
21+
- `docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d` -- takes 10-15 seconds
22+
23+
#### Compose Service (Neurosynth-Compose Backend + Frontend)
24+
- `cd compose && cp .env.example .env`
25+
- `docker compose build` -- takes 2-3 minutes. **NEVER CANCEL. Set timeout to 240+ seconds.**
26+
- `docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d` -- takes 5-10 seconds
27+
28+
### Database Setup
29+
Create and migrate databases for both services:
30+
31+
#### Store Database
32+
- `docker compose exec -T store-pgsql17 psql -U postgres -c "create database test_db"`
33+
- `docker compose exec -T store-pgsql17 psql -U postgres -d test_db -c "CREATE EXTENSION IF NOT EXISTS vector;"`
34+
- `docker compose exec -T neurostore bash -c "flask db merge heads && flask db stamp head && flask db migrate && flask db upgrade"` -- takes 5-10 seconds
35+
- `docker compose exec -T neurostore bash -c "flask ingest-neurosynth --max-rows 100"` -- takes 5-10 seconds
36+
37+
#### Compose Database
38+
- `docker compose exec -T compose_pgsql17 psql -U postgres -c "create database test_db"`
39+
- `docker compose exec -T compose bash -c "flask db merge heads && flask db stamp head && flask db migrate && flask db upgrade"` -- takes 5-10 seconds
40+
41+
### Frontend Development
42+
Navigate to `compose/neurosynth-frontend/`:
43+
44+
#### Build & Development
45+
- `cp .env.example .env.dev`
46+
- `npm install` -- takes 25-30 seconds
47+
- `npm run build:dev` -- takes 25-35 seconds
48+
- `npm run start:dev` -- starts development server on http://localhost:3000
49+
50+
#### Configuration Files Needed
51+
- `.env.dev` -- development environment configuration
52+
- `.env.staging` -- staging environment configuration
53+
54+
## Testing - Set Long Timeouts
55+
56+
### Backend Tests - NEVER CANCEL
57+
**CRITICAL**: Backend tests take significant time. Set appropriate timeouts.
58+
59+
#### Store Backend Tests
60+
- `docker compose run -e "APP_SETTINGS=neurostore.config.DockerTestConfig" --rm neurostore bash -c "python -m pytest neurostore/tests"`
61+
- Takes 2-3 minutes. **NEVER CANCEL. Set timeout to 300+ seconds.**
62+
- Expected: ~229 passed, 12 skipped
63+
64+
#### Compose Backend Tests
65+
- `docker compose run -e "APP_SETTINGS=neurosynth_compose.config.DockerTestConfig" --rm compose bash -c "python -m pytest neurosynth_compose/tests"`
66+
- Takes 2-4 minutes. **NEVER CANCEL. Set timeout to 360+ seconds.**
67+
68+
#### Frontend Tests
69+
- `npm run test` -- takes 50-70 seconds
70+
- Expected: ~345 tests passing
71+
- `npm run cy:e2e-headless` -- Cypress E2E tests (requires both backend services running)
72+
73+
## Validation Scenarios
74+
75+
### Always Test These Workflows After Changes
76+
1. **API Functionality**: `curl http://localhost/api/studies?limit=5` -- Store API should return JSON
77+
2. **Frontend Build**: Ensure `npm run build:dev` completes without errors
78+
3. **Backend Migration**: Test database migration commands work
79+
4. **Docker Services**: All containers should start and stay healthy
80+
81+
### Service URLs
82+
- **Store API**: http://localhost/api (port 80)
83+
- **Compose API**: http://localhost:81/api (port 81)
84+
- **Frontend Dev**: http://localhost:3000 (when running `npm run start:dev`)
85+
- **pgHero**: http://localhost/pghero (database monitoring)
86+
- **Grafana**: Available via docker compose (database metrics)
87+
88+
## Linting & Code Quality
89+
90+
### Backend Linting
91+
- `pip install flake8` (if not installed)
92+
- `cd store/backend && flake8 ./neurostore` -- takes <1 second
93+
- `cd compose/backend && flake8 ./neurosynth_compose` -- takes <1 second
94+
95+
### Frontend Linting
96+
- `npm run lint` -- takes 30-40 seconds
97+
- **Expected**: ~899 linting issues exist (mostly formatting). This is normal.
98+
- `npm run lint --fix` -- auto-fixes many issues
99+
100+
## Configuration Notes
101+
102+
### Environment Variables
103+
Both services use similar `.env` configurations:
104+
- `POSTGRES_HOST` -- Database host (store-pgsql17 or compose_pgsql17)
105+
- `POSTGRES_PASSWORD` -- Database password (usually "example")
106+
- `AUTH0_CLIENT_ID` -- Auth0 integration (can be placeholder for dev)
107+
- `DEBUG=True` -- Enable debug mode
108+
109+
### Docker Configuration Issues
110+
Development config uses `test_db` database by default, not the service-named databases. This is intentional for the `DevelopmentConfig` class.
111+
112+
## Directory Structure Reference
113+
```
114+
/
115+
├── store/ # Neurostore backend service
116+
│ ├── backend/ # Python Flask application
117+
│ │ ├── neurostore/ # Main package
118+
│ │ ├── migrations/ # Database migrations
119+
│ │ └── pyproject.toml # Python dependencies
120+
│ ├── docker-compose.yml # Production config
121+
│ └── docker-compose.dev.yml # Development overrides
122+
├── compose/ # Neurosynth-Compose service
123+
│ ├── backend/ # Python Flask application
124+
│ │ ├── neurosynth_compose/ # Main package
125+
│ │ └── pyproject.toml # Python dependencies
126+
│ ├── neurosynth-frontend/ # React/TypeScript app
127+
│ │ ├── src/ # Source code
128+
│ │ ├── package.json # Node.js dependencies
129+
│ │ └── vite.config.ts # Build configuration
130+
│ ├── docker-compose.yml # Production config
131+
│ └── docker-compose.dev.yml # Development overrides
132+
```
133+
134+
## Common Issues & Solutions
135+
136+
### Missing OpenAPI Specs Error
137+
- **Problem**: `FileNotFoundError: neurostore-openapi.yml` or `neurosynth-compose-openapi.yml`
138+
- **Solution**: Run `git submodule update --init --recursive`
139+
140+
### Missing TypeScript SDK Error
141+
- **Problem**: `Could not resolve "../neurostore-typescript-sdk"`
142+
- **Solution**: Initialize git submodules first
143+
144+
### Container Won't Start
145+
- **Problem**: `Worker failed to boot` or similar gunicorn errors
146+
- **Solution**: Check OpenAPI submodules are initialized and container logs with `docker compose logs [service]`
147+
148+
### Database Connection Errors
149+
- **Problem**: `database "test_db" does not exist`
150+
- **Solution**: Create test databases as shown in setup steps
151+
152+
### Frontend Build Sentry Warnings
153+
- **Problem**: Sentry API token errors during build
154+
- **Solution**: Expected in development. Build still succeeds - ignore Sentry warnings.
155+
156+
## Performance Expectations
157+
158+
| Operation | Time | Timeout |
159+
|-----------|------|---------|
160+
| Git submodules | 1-2s | 30s |
161+
| Store build | 1m36s | 180s |
162+
| Compose build | 1m56s | 240s |
163+
| Frontend npm install | 26s | 60s |
164+
| Frontend build | 29s | 90s |
165+
| Backend tests | 2m30s | 300s |
166+
| Frontend tests | 59s | 120s |
167+
| Database migrations | 5s | 30s |
168+
| Backend linting | <1s | 10s |
169+
| Frontend linting | 37s | 60s |
170+
171+
**Remember**: Always use `--timeout` parameters longer than these estimates. Builds may take longer on slower systems.

‎.github/workflows/monthly-docker-build.yml‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,16 @@ jobs:
1616
working-directory: store
1717
run: |
1818
echo "Services in store/docker-compose.yml:"
19-
docker-compose config --services
19+
cp .env.example .env
20+
docker compose config --services
2021
echo "Building all images in store/docker-compose.yml with --no-cache"
21-
docker-compose build --no-cache
22+
docker compose build --no-cache
2223
2324
- name: Build compose images
2425
working-directory: compose
2526
run: |
2627
echo "Services in compose/docker-compose.yml:"
27-
docker-compose config --services
28+
cp .env.example .env
29+
docker compose config --services
2830
echo "Building all images in compose/docker-compose.yml with --no-cache"
29-
docker-compose build --no-cache
31+
docker compose build --no-cache

‎.github/workflows/workflow.yml‎

Lines changed: 38 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,9 @@ permissions:
1717
jobs:
1818
clean:
1919
strategy:
20+
fail-fast: false
2021
matrix:
21-
package: ['neurosynth_compose', 'compose_nginx', 'compose_pgsql', 'compose_worker', 'neurostore', 'store_nginx', 'store-pgsql']
22+
package: ['neurosynth_compose', 'compose_nginx', 'compose_pgsql', 'compose_worker', 'neurostore', 'store_nginx', 'store-pgsql', 'store-pgsql17', 'compose_pgsql17']
2223
runs-on: ubuntu-latest
2324

2425
steps:
@@ -77,6 +78,10 @@ jobs:
7778
compose_pgsql.cache-from=type=registry,ref=ghcr.io/${{ github.repository_owner }}/compose_pgsql:${{ hashFiles('**/compose/postgres/**') }}
7879
compose_pgsql.cache-from=type=gha,scope=cached-compose-pgsql
7980
compose_pgsql.cache-to=type=gha,scope=cached-compose-pgsql,mode=max,ignore-error=true
81+
compose_pgsql17.tags=ghcr.io/${{ github.repository_owner }}/compose_pgsql17:${{ hashFiles('**/compose/postgres/**') }}
82+
compose_pgsql17.cache-from=type=registry,ref=ghcr.io/${{ github.repository_owner }}/compose_pgsql17:${{ hashFiles('**/compose/postgres/**') }}
83+
compose_pgsql17.cache-from=type=gha,scope=cached-compose-pgsql17
84+
compose_pgsql17.cache-to=type=gha,scope=cached-compose-pgsql17,mode=max,ignore-error=true
8085
compose_worker.tags=ghcr.io/${{ github.repository_owner }}/compose_worker:${{ hashFiles('**/compose/backend/neurosynth_compose/**') }}
8186
compose_worker.cache-from=type=registry,ref=ghcr.io/${{ github.repository_owner }}/compose_worker:${{ hashFiles('**/compose/backend/neurosynth_compose/**') }}
8287
compose_worker.cache-from=type=gha,scope=cached-compose-worker
@@ -130,6 +135,10 @@ jobs:
130135
store-pgsql.cache-from=type=registry,ref=ghcr.io/${{ github.repository_owner }}/store-pgsql:${{ hashFiles('**/store/postgres/**') }}
131136
store-pgsql.cache-from=type=gha,scope=cached-store-pgsql
132137
store-pgsql.cache-to=type=gha,scope=cached-store-pgsql,mode=max,ignore-error=true
138+
store-pgsql17.tags=ghcr.io/${{ github.repository_owner }}/store-pgsql17:${{ hashFiles('**/store/postgres/**') }}
139+
store-pgsql17.cache-from=type=registry,ref=ghcr.io/${{ github.repository_owner }}/store-pgsql17:${{ hashFiles('**/store/postgres/**') }}
140+
store-pgsql17.cache-from=type=gha,scope=cached-store-pgsql17
141+
store-pgsql17.cache-to=type=gha,scope=cached-store-pgsql17,mode=max,ignore-error=true
133142
134143
135144
neurostore_backend_tests:
@@ -165,20 +174,28 @@ jobs:
165174
neurostore.cache-from=type=gha,scope=cached-neurostore
166175
store_nginx.cache-from=type=gha,scope=cached-store-nginx
167176
store-pgsql.cache-from=type=gha,scope=cached-store-pgsql
177+
store-pgsql17.cache-from=type=gha,scope=cached-store-pgsql17
168178
-
169179
name: spin up backend
180+
id: spin_up_backend
181+
continue-on-error: true
170182
run: |
183+
set -o pipefail
171184
docker network create nginx-proxy
172185
docker compose pull --ignore-buildable
173186
docker compose \
174187
-f docker-compose.yml \
175188
-f docker-compose.dev.yml \
176189
up -d --no-build
190+
-
191+
name: Abort if compose failed
192+
if: ${{ steps.spin_up_backend.outcome == 'failure' }}
193+
run: exit 1
177194
-
178195
name: Create Test Database
179196
run: |
180197
docker compose exec -T \
181-
store-pgsql \
198+
store-pgsql17 \
182199
psql -U postgres -c "create database test_db"
183200
-
184201
name: Backend Tests
@@ -233,6 +250,7 @@ jobs:
233250
compose.cache-from=type=gha,scope=cached-compose
234251
compose_nginx.cache-from=type=gha,scope=cached-compose-nginx
235252
compose_pgsql.cache-from=type=gha,scope=cached-compose-pgsql
253+
compose_pgsql17.cache-from=type=gha,scope=cached-compose-pgsql17
236254
compose_worker.cache-from=type=gha,scope=cached-compose-worker
237255
-
238256
name: Spin up backend
@@ -247,7 +265,7 @@ jobs:
247265
name: Create Test Database
248266
run: |
249267
docker compose exec -T \
250-
compose_pgsql \
268+
compose_pgsql17 \
251269
psql -U postgres -c "create database test_db"
252270
-
253271
name: Backend Tests
@@ -323,6 +341,7 @@ jobs:
323341
compose.cache-from=type=gha,scope=cached-compose
324342
compose_nginx.cache-from=type=gha,scope=cached-compose-nginx
325343
compose_pgsql.cache-from=type=gha,scope=cached-compose-pgsql
344+
compose_pgsql17.cache-from=type=gha,scope=cached-compose-pgsql17
326345
compose_worker.cache-from=type=gha,scope=cached-compose-worker
327346
-
328347
name: Spin up backend
@@ -337,7 +356,7 @@ jobs:
337356
name: Create Test Database
338357
run: |
339358
docker compose exec -T \
340-
compose_pgsql \
359+
compose_pgsql17 \
341360
psql -U postgres -c "create database test_db"
342361
-
343362
name: Frontend Jest Unit Tests
@@ -405,6 +424,7 @@ jobs:
405424
compose.cache-from=type=gha,scope=cached-compose
406425
compose_nginx.cache-from=type=gha,scope=cached-compose-nginx
407426
compose_pgsql.cache-from=type=gha,scope=cached-compose-pgsql
427+
compose_pgsql17.cache-from=type=gha,scope=cached-compose-pgsql17
408428
compose_worker.cache-from=type=gha,scope=cached-compose-worker
409429
-
410430
name: load store images
@@ -419,6 +439,7 @@ jobs:
419439
neurostore.cache-from=type=gha,scope=cached-neurostore
420440
store_nginx.cache-from=type=gha,scope=cached-store-nginx
421441
store-pgsql.cache-from=type=gha,scope=cached-store-pgsql
442+
store-pgsql17.cache-from=type=gha,scope=cached-store-pgsql17
422443
-
423444
name: Spin up compose backend
424445
run: |
@@ -431,29 +452,40 @@ jobs:
431452
up -d --no-build
432453
-
433454
name: Spin up store backend
455+
id: spin_up_store_backend
456+
continue-on-error: true
434457
run: |
458+
set -o pipefail
435459
cd store
436460
docker compose pull --ignore-buildable
437461
docker compose \
438462
-f docker-compose.yml \
439463
-f docker-compose.dev.yml \
440464
up -d --no-build
465+
-
466+
name: Abort if store compose failed
467+
if: ${{ steps.spin_up_store_backend.outcome == 'failure' }}
468+
run: exit 1
441469
-
442470
name: Create Compose Database
443471
run: |
444472
cd compose
445473
446474
docker compose exec -T \
447-
compose_pgsql \
475+
compose_pgsql17 \
448476
psql -U postgres -c "create database test_db"
449477
-
450478
name: Create Store Database
451479
run: |
452480
cd store
453481
454482
docker compose exec -T \
455-
store-pgsql \
483+
store-pgsql17 \
456484
psql -U postgres -c "create database test_db"
485+
486+
docker compose exec -T \
487+
store-pgsql17 \
488+
psql -U postgres -d test_db -c "CREATE EXTENSION IF NOT EXISTS vector;"
457489
-
458490
name: Initialize Compose Database
459491
run: |

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
.neurostuff_secrets
12
historical/
23

34
# node

0 commit comments

Comments
 (0)