Skip to content

Commit b4b3e77

Browse files
shnizzedyshnizzedy
authored
✨ Curious alerts to REDCap (#8)
2 parents bd92885 + cdbb0b2 commit b4b3e77

51 files changed

Lines changed: 3000 additions & 1022 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,16 @@ terraform.rc
2727
generated_*
2828
**/generated_*
2929

30+
# Keep backups in git (they're important!)
31+
!backups/
32+
!backups/*.tfstate.*
33+
34+
# Lockfiles
35+
.lock
36+
**/.terraform.lock
37+
**/terraform.lock
38+
!**/.terraform.lock.hcl
39+
3040
# Linting temporary files
3141
.lint_temp
3242
**/.lint_temp/

‎.gitmodules‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "javascript_jobs/autoexport"]
2+
path = javascript_jobs/autoexport
3+
url = https://github.com/childmindresearch/mindlogger-autoexport.git

‎.pre-commit-config.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,16 @@ repos:
2424
rev: v1.105.0
2525
hooks:
2626
- id: terraform_fmt
27+
files: ^infrastructure/
2728
- id: terraform_tflint
2829
args:
2930
- --init
3031
- --config=__GIT_WORKING_DIR__/.tflint.hcl
32+
files: ^infrastructure/
3133
- id: terraform_validate
34+
files: ^infrastructure/
3235
- id: terraform_checkov
36+
files: ^infrastructure/
3337
- repo: https://github.com/DavidAnson/markdownlint-cli2
3438
rev: v0.21.0
3539
hooks:

‎CHANGELOG.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111

1212
- initial Terraform configuration
1313
- utility function library
14-
- Ripple to REDCap Python script / job
14+
- Python scripts / jobs
15+
- Ripple to REDCap
16+
- REDCap to REDCap
17+
- REDCap to Curious
18+
- Curious alerts to REDCap

‎VERSION‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
1.2.0-dev
1+
1.3.0-dev

‎infrastructure/.terraform.lock.hcl‎

Lines changed: 0 additions & 21 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎infrastructure/README.md‎

Lines changed: 266 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,266 @@
1+
# HBN Migration Infrastructure
2+
3+
Terraform configuration for managing HBN Migration systemd services on the VM.
4+
5+
## Services Managed
6+
7+
- **ripple-to-redcap**: Syncs data from Ripple to REDCap
8+
- **redcap-to-redcap**: Syncs REDCap data
9+
- **redcap-to-curious**: Syncs data from REDCap to Curious
10+
- **curious-alerts-websocket**: Always-on WebSocket service for Curious alerts
11+
- **hbn-sync.timer**: Runs the above sync services on schedule
12+
13+
## Quick Start
14+
15+
### 1. Initial Setup
16+
17+
```bash
18+
# Install Terraform and dependencies
19+
./setup_terraform.sh
20+
21+
# Copy and configure variables
22+
cp terraform.tfvars.example terraform.tfvars
23+
vim terraform.tfvars # Edit with your values
24+
25+
# Initialize Terraform
26+
terraform init
27+
```
28+
29+
### 2. Deploy Services
30+
31+
```bash
32+
./safe-apply.sh
33+
```
34+
35+
### 3. Verify Deployment
36+
37+
```bash
38+
# Check services
39+
sudo systemctl status curious-alerts-websocket.service
40+
sudo systemctl status hbn-sync.timer
41+
sudo systemctl list-timers
42+
43+
# View logs
44+
sudo journalctl -u curious-alerts-websocket -f
45+
tail -f /var/log/hbnmigration/*.log
46+
```
47+
48+
## Workspaces
49+
50+
This infrastructure supports Terraform workspaces for isolated testing.
51+
52+
### Available Workspaces
53+
54+
- `default` - Production environment
55+
- `test` - Testing environment (or create your own)
56+
57+
### Workspace Commands
58+
59+
```bash
60+
# Create and switch to test workspace
61+
terraform workspace new test
62+
63+
# List workspaces
64+
terraform workspace list
65+
66+
# Switch between workspaces
67+
terraform workspace select test
68+
terraform workspace select default
69+
70+
# Apply to current workspace
71+
terraform apply
72+
```
73+
74+
### Workspace Isolation
75+
76+
Each workspace maintains:
77+
78+
- Separate state files (`terraform.tfstate.d/<workspace>/`)
79+
- Isolated service names (e.g., `test-ripple-to-redcap.service`)
80+
- Separate project directories (e.g., `/opt/hbnmigration-test`)
81+
- Isolated log directories (e.g., `/var/log/hbnmigration-test`)
82+
- Independent cron jobs for state backups
83+
84+
### Testing Workflow
85+
86+
```bash
87+
# 1. Create test workspace
88+
terraform workspace new test
89+
90+
# 2. Deploy test services
91+
terraform apply
92+
93+
# 3. Verify test services
94+
sudo systemctl status test-ripple-to-redcap.service
95+
sudo journalctl -u test-curious-alerts-websocket -f
96+
97+
# 4. When satisfied, deploy to production
98+
terraform workspace select default
99+
terraform apply
100+
```
101+
102+
## Configuration
103+
104+
Edit `terraform.tfvars` to customize:
105+
106+
- `project_root` - Base directory for the application
107+
- `user_group` - System user/group for running services
108+
- `sync_interval_minutes` - How often sync services run (1-1440)
109+
- `venv_path` - Virtual environment location (relative or absolute)
110+
- `log_directory` - Where logs are stored
111+
112+
## State Management
113+
114+
### Automatic Backups
115+
116+
State backups run automatically:
117+
118+
- **Location**: `/var/log/hbnmigration/terraform-backups/` (or `/var/log/hbnmigration-<workspace>/terraform-backups/`)
119+
- **Retention**: 30 timestamped backups, 7 daily backups
120+
- **Schedule**: Daily via cron
121+
- **Manual backup**:
122+
123+
```bash
124+
./backup-state.sh
125+
```
126+
127+
### Restore from Backup
128+
129+
```bash
130+
# List backups
131+
ls -lh /var/log/hbnmigration/terraform-backups/
132+
133+
# Restore (make sure you're in the correct workspace!)
134+
terraform workspace select default # or test
135+
cp /var/log/hbnmigration/terraform-backups/terraform.tfstate.YYYYMMDD-HHMMSS \
136+
terraform.tfstate
137+
138+
# Verify
139+
terraform state list
140+
```
141+
142+
## Service Configuration
143+
144+
All services receive a `WORKSPACE` environment variable indicating which workspace deployed them. Use this in your Python code to load workspace-specific configurations:
145+
146+
```python
147+
import os
148+
149+
workspace = os.environ.get('WORKSPACE', 'default')
150+
config_file = f'config.{workspace}.yaml'
151+
```
152+
153+
## Monitoring Services
154+
155+
### Default Workspace
156+
157+
```bash
158+
# Service status
159+
sudo systemctl status ripple-to-redcap.service
160+
sudo systemctl status redcap-to-redcap.service
161+
sudo systemctl status redcap-to-curious.service
162+
sudo systemctl status curious-alerts-websocket.service
163+
164+
# Live logs
165+
sudo journalctl -u curious-alerts-websocket -f
166+
sudo journalctl -u ripple-to-redcap -u redcap-to-redcap -u redcap-to-curious -f
167+
168+
# File logs
169+
tail -f /var/log/hbnmigration/*.log
170+
```
171+
172+
### Test Workspace
173+
174+
```bash
175+
# Service status
176+
sudo systemctl status test-ripple-to-redcap.service
177+
sudo systemctl status test-curious-alerts-websocket.service
178+
179+
# Live logs
180+
sudo journalctl -u test-curious-alerts-websocket -f
181+
sudo journalctl -u test-ripple-to-redcap -f
182+
183+
# File logs
184+
tail -f /var/log/hbnmigration-test/*.log
185+
```
186+
187+
## Cleaning Up Workspaces
188+
189+
To remove a test workspace:
190+
191+
```bash
192+
# 1. Stop and disable services
193+
sudo systemctl stop test-*.service test-*.timer
194+
sudo systemctl disable test-*.service test-*.timer
195+
196+
# 2. Remove service files
197+
sudo rm /etc/systemd/system/test-*.service
198+
sudo rm /etc/systemd/system/test-*.timer
199+
sudo rm /etc/systemd/system/test-*.target
200+
sudo systemctl daemon-reload
201+
202+
# 3. (Optional) Remove workspace directories and logs
203+
sudo rm -rf /opt/hbnmigration-test
204+
sudo rm -rf /var/log/hbnmigration-test
205+
206+
# 4. Delete workspace
207+
terraform workspace select default
208+
terraform workspace delete test
209+
```
210+
211+
## Maintenance
212+
213+
### Update Service Configuration
214+
215+
1. Edit service templates in `./services/*.tpl`
216+
2. Run `./safe-apply.sh`
217+
3. Services are automatically restarted
218+
219+
### Change Sync Interval
220+
221+
1. Edit `sync_interval_minutes` in `terraform.tfvars`
222+
2. Run `./safe-apply.sh`
223+
3. Timer is automatically updated
224+
225+
### View All Service Logs
226+
227+
```bash
228+
# WebSocket service (always-on)
229+
sudo journalctl -u curious-alerts-websocket -f
230+
231+
# Sync services (triggered by timer)
232+
sudo journalctl -u ripple-to-redcap -u redcap-to-redcap -u redcap-to-curious -f
233+
234+
# All file logs
235+
tail -f /var/log/hbnmigration/*.log
236+
237+
# Check timer schedule
238+
sudo systemctl list-timers hbn-sync.timer
239+
```
240+
241+
## Troubleshooting
242+
243+
### Services won't start
244+
245+
```bash
246+
# Check service status
247+
sudo systemctl status <service-name>
248+
249+
# View full logs
250+
sudo journalctl -u <service-name> -n 100
251+
252+
# Check permissions
253+
ls -la /opt/hbnmigration
254+
ls -la /var/log/hbnmigration
255+
```
256+
257+
### Wrong workspace deployed
258+
259+
```bash
260+
# Check current workspace
261+
terraform workspace show
262+
263+
# View deployed services
264+
systemctl list-units 'test-*.service' --all
265+
systemctl list-units '*ripple*.service' --all
266+
```

‎infrastructure/backup-state.sh‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/bin/bash
2+
set -e
3+
4+
STATE_DIR="/opt/hbnmigration/terraform"
5+
BACKUP_DIR="/opt/hbnmigration/terraform/backups"
6+
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
7+
8+
echo "Backing up Terraform state..."
9+
10+
# Create backup directory
11+
mkdir -p "$BACKUP_DIR"
12+
13+
# Backup state file
14+
if [ -f "$STATE_DIR/terraform.tfstate" ]; then
15+
cp "$STATE_DIR/terraform.tfstate" "$BACKUP_DIR/terraform.tfstate.$TIMESTAMP"
16+
echo "✓ State backed up to: $BACKUP_DIR/terraform.tfstate.$TIMESTAMP"
17+
18+
# Keep only last 30 backups
19+
cd "$BACKUP_DIR"
20+
find . -maxdepth 1 -name "terraform.tfstate.*" -type f -printf '%T@ %p\n' | \
21+
sort -rn | \
22+
tail -n +31 | \
23+
cut -d' ' -f2- | \
24+
xargs -r rm
25+
echo "✓ Cleaned up old backups (keeping last 30)"
26+
else
27+
echo "⚠️ No state file found at $STATE_DIR/terraform.tfstate"
28+
fi
29+
30+
# Backup to git if in a git repo
31+
if git -C "$STATE_DIR" rev-parse --git-dir > /dev/null 2>&1; then
32+
cd "$STATE_DIR"
33+
git add "backups/terraform.tfstate.$TIMESTAMP" 2>/dev/null || true
34+
echo "✓ State backup staged for git commit"
35+
fi

‎infrastructure/backups/.gitkeep‎

Whitespace-only changes.

0 commit comments

Comments
 (0)