Implement persistent storage for PSQL (#746)

* Docker: Persist PSQL Data in Volumes

* Docker: env loaded in compose + docs

* Docker: Persistent Data Final Review

Author: kyleecodes

README.md

@@ -19,8 +19,6 @@ Chayn's Bloom service offers several key features designed to support individual
 
 ## Bloom Backend Technical Documentation
 
-Read our [Bloom Backend Tech Wiki Docs](https://github.com/chaynHQ/bloom-backend/wiki) for overviews of key concepts and data & database architecture.
-
 Technologies Used:
 
 - [NestJS](https://nestjs.com/) - NodeJs framework for building scalable and reliable server-side applications
@@ -38,7 +36,9 @@ Technologies Used:
 - [GitHub Actions](https://github.com/features/actions) - CI pipeline
 - [ESLint](https://eslint.org/) and [Prettier](https://prettier.io/) for linting and formatting.
 
-## Local Development
+Read our [Bloom Backend Tech Wiki Docs](https://github.com/chaynHQ/bloom-backend/wiki) for overviews of key concepts and data & database architecture.
+
+## Local Development Directions:
 
 Making an open-source contribution you have agreed to our [Code of Conduct](/CODE_OF_CONDUCT.md).
 

docker-compose.yml

@@ -6,7 +6,10 @@ services:
     build:
       context: .
       dockerfile: Dockerfile
+    env_file: '.env'
     environment:
+      # To connect to local your local psql db, replace DATABASE_URL with:
+      # postgres://postgres:[email protected]:5432/bloom
       DATABASE_URL: postgres://postgres:postgres@db:5432/bloom
     ports:
       - 35001:35001
@@ -25,3 +28,8 @@ services:
       POSTGRES_USER: postgres
       POSTGRES_PASSWORD: postgres
       POSTGRES_DB: bloom
+    volumes:
+      - postgres-data:/var/lib/postgresql/data # path for named volume
+
+volumes:
+  postgres-data: # named volume for persistent postgres data

docs/database-guide.md

@@ -2,65 +2,92 @@
 
 ## How to Populate the Database
 
-**Prerequisites:**
+### Prerequisites
 
-- [Postgres 16](https://www.postgresql.org/download/) \*technically not required if running in Docker
-- Running Bloom’s backend
+- Bloom's backend is running
 
 ### Summary
 
-Most open-source contributions (like running Cypress integration tests from the frontend) require adding test data to your local database. To do this, download Bloom's test data dump file, connect to the database server, then populate the database with the backup data.
+Populating your database with test data is essential for a fully functional development environment, making full-stack contributions, and running Cypress integration tests from the frontend. However, it is not necessary for smaller, isolated contributions.
+
+First, download Bloom's test data dump file. Next, connect to the database server, restore your database with the dump file, then verify with a query.
 
 ### Download Test Data Dump File
 
-Download the test data dump file [linked here from our Google Drive](https://drive.google.com/file/d/1y6KJpAlpozEg3GqhK8JRdmK-s7uzJXtB/view?usp=drive_link).
+First, download the test data dump file [linked here from our Google Drive](https://drive.google.com/file/d/1y6KJpAlpozEg3GqhK8JRdmK-s7uzJXtB/view?usp=drive_link). Then place this dump file in the project directory.
 
 ### Connect to Server and Add Data
 
-There are multiple methods you can use here. For simplicity, these directions assume you are using Docker.
+Next, connect to the database server and add test data from the dump file, using the appropriate commands based on how you are running the app - fully containerized, containerized app with local database, or manually without Docker.
 
-Run the following command to restore the database from the dump file using pg_restore:
+1. Restore the database from the dump file by running these pg_restore commands.
 
-```
-docker exec -i <container_name> pg_restore -U <username> -d <database_name> --clean --if-exists < </path/to/dumpfile.dump>
-```
+   **Fully Containerized App Command:**
 
-`container_name`, `username`, and `database_name` are defined in the `docker-compose.yml` file under ‘db’.
+   ```
+   docker exec -i <container_name> pg_restore -U <username> -d <database_name> --clean --if-exists < /path/to/dumpfile.dump
+   ```
 
-Start the bloom psql database server:
+   `container_name`, `username`, and `database_name` are defined in the `docker-compose.yml` under the ‘db’ service. Here is the same command with the default values:
 
-```
-docker exec -it <container_name> psql -U <username> -d <database_name>
-```
+   ```
+   docker exec -i bloom-local-db pg_restore -U postgres -d bloom --clean --if-exists < /path/to/dumpfile.dump
+   ```
 
-This will open the psql server for bloom, where you can run queries to verify the restore.
+   **Docker with Local DB or Running Manually Command:**
 
-You can verify the restore by running a SQL query to test if one of our test user's data has been properly populated into the database:
+   ```
+   pg_restore -U postgres -d bloom --clean --if-exists /path/to/dumpfile.dump
+   ```
 
-```
-SELECT * FROM public."user" users WHERE users."email" = '[email protected]';
-```
+2. Next, start the bloom psql database server.
 
-If the user exists, the database has successfully been seeded!
+   **Fully Containerized App Command:**
+
+   ```
+   docker exec -it <container_name> psql -U <username> -d <database_name>
+
+   # same command with default values added:
+   docker exec -it bloom-local-db psql -U postgres -d bloom
+   ```
+
+   **Docker with Local DB or Running Manually Command:**
+
+   ```
+   psql -U <username> -h localhost -p 5432 -d <database_name>
+   ```
+
+3. Verify the restore by running queries in the psql server.
+
+   ```
+   SELECT \* FROM public."user" users WHERE users."email" = '[email protected]';
+   ```
+
+   If the user exists, your database has successfully been populated with test data!
 
 ### Troubleshooting
 
+- Persistent storage is configured in the `docker-compose.yml` file using [named volumes](https://docs.docker.com/engine/storage/volumes/). This maintains your data, even if you delete your container. If you have issues with accessing persistent db storage, try replacing the volume path with an absolute path, or update your firewall settings if using WSL (especially if running integration tests). If issues with volumes persist, remove the named volumes from `docker-compose.yml` and populate your database manually as needed.
+- Ensure both the 'db' and 'api' containers are running.
+- Hard reset Docker containers `docker-compose up -d --force-recreate`.
 - If you remove **`--clean`** from the restore command but encounter duplicate object errors, the existing schema may conflict with the restore. In that case, clean the specific objects manually or use **`DROP SCHEMA public CASCADE`** before restoring.
-- Verify that the dump file is valid by running: `pg_restore --list yourfile.dump` If it fails to list contents, the dump file may be corrupted or incomplete.
-- In the psql server, verify the tables and columns exist with `\dt` , `\dt public.*` , and `\d public."user";`
+- Verify that the dump file is valid by running: `pg_restore --list yourfile.dump` If it fails to list contents, the dump file may be corrupted or incomplete. Please notify our team if this happens.
+- Verify the tables and columns exist within the psql server with `\dt` , `\dt public.*` , and `\d public."user";`
 - Run a **`DROP SCHEMA`** or truncate tables before running **`pg_restore`:**
+
   ```
   DROP SCHEMA public CASCADE;
   CREATE SCHEMA public;
   ```
-- Try the following: delete the existing db, create a new db with the same name, and try the restore on this new db. The db drop may throw an error, if so run the following command first.
-
-  `SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = 'bloom';`
 
+- To hard reset the database in the psql server, first delete the existing db, then create a new db with the same name, and try the restore on this new db. The db drop may throw an error, if so run the following command first:
+  ```
+  SELECT pg_terminate_backend(pid) FROM      pg_stat_activity WHERE datname = 'bloom';`
+  ```
   Then drop the database using:
-
-  `DROP DATABASE bloom;`
-
+  ```
+  DROP DATABASE bloom;
+  ```
 - If the sql dump file is outdated, you can update it by running `docker compose down` then `docker compose up` again as this is configured to run migrations.
 
 ### Chayn Staff - Heroku Directions
@@ -72,11 +99,11 @@ Chayn staff with access to Heroku, you also have the option to seed the database
 3. Replace <HEROKU_APP_NAME> with the correct Heroku app name in the `seed-local-db.sh file`
 4. Run `chmod +x ./seed-local-db.sh` in your terminal to make the file executable
 
-   After the above has been confirmed, run
+After the above has been confirmed, run
 
-   ```bash
-   bash seed-local-db.sh
-   ```
+```bash
+bash seed-local-db.sh
+```
 
 ## Database Migrations
 

docs/local-development.md

@@ -2,35 +2,26 @@
 
 ## Summary
 
-**The develop branch is our source of truth.** Fork from develop, create new feature branch, then when your PR is merged, develop will automatically merge into the main branch for deployment to production.
-
 To run Bloom's backend:
 
 1. Install prerequisites
 2. Configure environment variables
 3. Install dependencies
-4. Run in a Dev Container, with Docker, or manually.
-5. Populate the database (required for most fullstack contributions and running integration tests from the frontend)
+4. Run the app using Docker, Dev Containers, or Manually
+5. Populate the database
 
 To test the backend:
 
 - Run unit tests
-- Run e2e integration tests from the frontend for fullstack contributions \*requires populating the database with data first
+- Run e2e integration tests from the frontend for full-stack contributions
 
 ## Prerequisites
 
 - NodeJS v20.x
 - Yarn v1.x
-- Docker
-- PostgreSQL 16
-
-#### Recommended Minimum System Requirements:
+- Docker and / or PostgreSQL
 
-- CPU: Quad-core 2.5 GHz (i5/Ryzen 5)
-- Memory: 16 GB RAM
-- Storage: 512 GB
-- OS: Linux, macOS, Windows, or WSL2 (latest versions)
-- Internet Connection: For accessing dependencies and external APIs/services
+_Recommended Minimum System Requirements: CPU: Quad-core 2.5 GHz (i5/Ryzen 5), Memory: 16 GB RAM, Storage: 512 GB, OS: Linux, macOS, Windows, or WSL2 (latest versions), Internet Connection: For accessing dependencies and external APIs/services._
 
 ## Configure Environment Variables
 
@@ -46,15 +37,17 @@ yarn
 
 There are 3 methods you can use to run Bloom’s backend locally:
 
-1. **Using Docker (recommended)** - runs postgres in a container.
-2. **Visual Studio Code Dev Container (recommended for Visual Studio users)** - installs all dependencies and the postgres database container automatically.
-3. **Manually** - manage postgres locally.
+1. **Using Docker (recommended)** - the backend app is fully containerized, installing PostgreSQL is optional.
+2. **Visual Studio Code Dev Container (recommended for Visual Code users)** - installs all dependencies and the PostgreSQL database container automatically.
+3. **Manually (recommended for PostgreSQL users)** - run the app with yarn and manage PostgreSQL locally.
 
-### With Docker - Recommended
+### Run with Docker - Recommended
 
-Bloom's backend is containerized and can be run solely in Docker - both the PostgreSQL database and NestJS app. This uses the least resources on your computer. To run the backend locally, make sure your system has Docker installed - you may need Docker Desktop if using a Mac or Windows.
+Prequisites: Docker (we recommend [Docker Desktop](https://docs.docker.com/desktop/)), PostgreSQL (optional).
 
-First, make sure the Docker app is running (just open the app). Then run
+Bloom's backend is fully containerized - both PostgreSQL and NestJS app. This does not require PostgreSQL to be installed locally. To connect to a local PostgreSQL database instead, modify the `DATABASE_URL` in the `docker-compose.yml` file. This will enable communications between Docker and your local database.
+
+To start the Docker container run:
 
 ```bash
 docker-compose up
@@ -66,9 +59,7 @@ You should see this in the shell output:
 Listening on localhost:35001, CTRL+C to stop
 ```
 
-_Note: you can use an application like Postman to test the apis locally_
-
-### Run in Dev Container - Recommended for Visual Studio Users
+### Run with Dev Container - Recommended for Visual Studio Users
 
 This method will automatically install all dependencies, IDE settings, and postgres container in a Dev Container (Docker container) within Visual Studio Code.
 
@@ -88,9 +79,13 @@ The dev Container is configured in the `.devcontainer` directory:
 
 See [Visual Studio Code Docs: Developing Inside a Dev Container](https://code.visualstudio.com/docs/devcontainers/containers) for more info.
 
-### Run Manually
+### Run Manually - Recommended for PostgreSQL Users
 
-Manage postgres locally to [populate the database](#populate-database), then run:
+Prerequisites: PostgreSQL
+
+Log into PostgreSQL and create a database called "bloom". Ensure it is running on port `35000` (or your desired port). Finally, start the PostgreSQL server on your machine.
+
+With the psql server running, start the app:
 
 ```bash
 yarn start:dev
@@ -148,16 +143,16 @@ See the [database-guide.md](database-guide.md) for instructions.
 
 # Git Flow and Deployment
 
-**The develop branch is our source of truth, not main.**
+**The develop branch is our source of truth, not main.** Fork from `develop`, create new feature branch, then when your PR is merged, `develop` will automatically merge into the main branch for deployment to production. Keep your branch updated by rebasing and merging feature/bug branches into `develop` as you code.
 
-Create new branches from the `develop` base branch. There is no need to run the build command before pushing changes to GitHub, simply push and create a pull request for the new branch. GitHub Actions will run build and linting tasks automatically. Rebase and merge feature/bug branches into `develop`.
-
-This will trigger an automatic deployment to the staging app by Heroku.
+Once your PR is merged to `develop`, this will trigger an automatic deployment to the staging app by Heroku.
 
 When changes have been tested in staging, merge `develop` into `main`. This will trigger an automatic deployment to the production app by Heroku.
 
-# Swagger
+# APIs
 
 Swagger automatically reflects all of the endpoints in the app, showing their urls and example request and response objects.
 
 To access Swagger simply run the project and visit http://localhost:35001/swagger
+
+For testing APIs, we recommend using tools like Postman.