Skip to content

Some fixes to prepare for production #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
davenquinn merged 11 commits into from
Apr 27, 2025
Merged

Some fixes to prepare for production #248

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
9b03a25
Remove old credentials files
davenquinn Apr 27, 2025
0779ec8
Add API version string to /meta route
davenquinn Apr 27, 2025
dedea13
Add release generation script
davenquinn Apr 27, 2025
dad6666
Updated credential handling
davenquinn Apr 27, 2025
461f617
Added info to /meta route
davenquinn Apr 27, 2025
b87a7e4
Add deprecation notice for v1
davenquinn Apr 27, 2025
ec68b85
Formatted with prettier
davenquinn Apr 27, 2025
51c1c8a
Remove unused settings files
davenquinn Apr 27, 2025
6a90730
Updated version number
davenquinn Apr 27, 2025
51640c1
Updated yarn version
davenquinn Apr 27, 2025
43fd0b1
Replace most console.log with larkin.trace (which can be silenced in …
davenquinn Apr 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
MACROSTRAT_DATABASE=<database_url>
# Useful for development
NODE_TLS_REJECT_UNAUTHORIZED=1

# Optional database locations
# - these are inferred automatically from MACROSTRAT_DATABASE if not set
# - if they are not found, some routes will not work
ELEVATION_DATABASE=<database_url>
ALICE_DATABASE=<database_url>
ROCKD_DATABASE=<database_url>
WHOS_ON_FIRST_DATABASE=<database_url>
21 changes: 8 additions & 13 deletions .github/workflows/build-image.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,19 @@ on:
push:
branches: ["main"]
tags:
- 'v*.*.*' # glob for semver tags (including prereleases)
- "v*.*.*" # glob for semver tags (including prereleases)
pull_request:
branches: [main]

jobs:
docker:
runs-on: ubuntu-latest
steps:
-
name: Checkout
- name: Checkout
uses: actions/checkout@v3
with:
submodules: 'recursive'
-
name: Docker meta
submodules: "recursive"
- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
Expand All @@ -34,24 +32,21 @@ jobs:
type=semver,pattern={{version}},suffix=-{{date 'YYYYMMDDHHmmss'}}
flavor: |
latest=false
-
name: Set up Docker Buildx
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

-
name: Login to OSG DockerHub
- name: Login to OSG DockerHub
uses: docker/login-action@v2
with:
registry: hub.opensciencegrid.org
username: ${{ vars.HARBOR_CLI_NAME }}
password: ${{ secrets.HARBOR_CLI_SECRET }}
-
name: Build and push
- name: Build and push
uses: docker/build-push-action@v3
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
cache-to: type=gha,mode=max
7 changes: 3 additions & 4 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,9 @@ npm-debug.log
node_modules
v1/node_modules/*
v1/npm-debug.log
v1/utilities/scripts/credentials.py
.idea
venv

!v1/credentials.ts
v2/utilities/scripts/credentials.py
v1/credentials.ts

.DS_Store
.AppleDouble
Expand All @@ -31,3 +28,5 @@ Icon
.Spotlight-V100
.Trashes
.yarn/install-state.gz

.env
2 changes: 2 additions & 0 deletions .prettierignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.yarn
node_modules
10 changes: 0 additions & 10 deletions .vscode/settings.json
View file
Open in desktop

This file was deleted.

893 changes: 0 additions & 893 deletions .yarn/releases/yarn-4.0.2.cjs
View file
Open in desktop

This file was deleted.

948 changes: 948 additions & 0 deletions .yarn/releases/yarn-4.9.1.cjs
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .yarnrc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
nodeLinker: node-modules

yarnPath: .yarn/releases/yarn-4.0.2.cjs
yarnPath: .yarn/releases/yarn-4.9.1.cjs
10 changes: 3 additions & 7 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,16 @@
# We have to specify a new-ish version of Debian
# to make sure we don't use an out of date image
FROM node:14-bullseye
FROM node:20-bullseye

RUN apt-get -y update && \
apt-get -y install redis-tools postgresql-client && \
apt-get -y install ca-certificates supervisor

WORKDIR /code

#COPY ./v1/package.json /code/v1/
#COPY ./v2/package.json /code/v2/
COPY ./package.json package.json
COPY ./package.json ./yarn.lock ./code/

RUN npm install
#RUN npm --prefix v1 install
RUN npm --prefix v2 install
RUN yarn install

# We had to switch to native postgres bindings
# (using the pg-native module) for the purposes
Expand Down
10 changes: 10 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,19 @@ DOCKER_TAG = latest

.PHONY: build run stop clean

# Get the current version from package.json
VER := $(shell node -p "require('./package.json').version")

run:
docker build -t macrostrat-api .
docker run --env-file .env --rm -it -p 5000:5000 macrostrat-api

release:
# Ensure that the repository is clean
git diff-index --quiet HEAD --
git tag -a v$(VER) -m "Version $(VER)"
git push origin v$(VER)

format:
# Format the code using Prettier
yarn run prettier --write .
53 changes: 35 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Macrostrat API

The API for SCIENCE

## About

The Macrostrat API provides diverse macrostratigraphic and geologic data in a
Expand All @@ -12,12 +12,13 @@ concise, queryable format.
#### Software

MariaDB, PostgreSQL, Nodejs, and the
[Macrostrat system](https://github.com/UW-Macrostrat/macrostrat) are required. Redis is
optional. When the API starts requests for all columns and all units are cached.
If Redis is available it will be used, otherwise they are cached in the
application memory.
[Macrostrat system](https://github.com/UW-Macrostrat/macrostrat) are required.
Redis is optional. When the API starts requests for all columns and all units
are cached. If Redis is available it will be used, otherwise they are cached in
the application memory.

The API requires `node` v14 or later to run. Recent testing has been conducted on `node` v18.
The API requires `node` v14 or later to run. Recent testing has been conducted
on `node` v18.

#### Data

Expand Down Expand Up @@ -47,21 +48,38 @@ NOTE: Postgres connections are completely broken in node v14 and v15 (as of
This will also run the script `postinstall.sh` which copies credentials files
into place.

## Running the API

Add environment variables to your shell or `.env` file. The
`MACROSTRAT_DATABASE_URL` variable is required.

To start the API run `npm start <port|5000>`. This will start the API on the
specified port (default 5000).

## Generating a release

**Old config below. We now use env variables specifying which db and environment to connect to.**
For `credentials.ts` fill in your MariaDB and PostgreSQL user information,
update the port for Redis if necessary, and follow the inline instructions for
generating a cache refresh key. The cache refresh key is used as a secret
parameter to programmatically refresh the column cache, such as in situations in
which the underlying data has been changed (editing, adding, etc).
Update the version string in `package.json` and run:

```
make release
```

This will simply add a tag that can be picked up by the CI/CD system to deploy
the application.

**Old config below. We now use env variables specifying which db and environment
to connect to.** For `credentials.ts` fill in your MariaDB and PostgreSQL user
information, update the port for Redis if necessary, and follow the inline
instructions for generating a cache refresh key. The cache refresh key is used
as a secret parameter to programmatically refresh the column cache, such as in
situations in which the underlying data has been changed (editing, adding, etc).

## Running

To start the API simply run `node server.js` which will start a process on
port 5050. Note that the default port was changed from 5000 to 5050 to avoid a
port conflict introduced in recent versions of commonly used OSs. To use a
different port, you can specify it during startup as so: `node server.js 5151`.
To start the API simply run `npm start` which will start a process on port 5050.
Note that the default port was changed from 5000 to 5050 to avoid a port
conflict introduced in recent versions of commonly used OSs. To use a different
port, you can specify it during startup as so: `node server.js 5151`.

For production use `pm2` is recommended. To start as a single process:

Expand Down Expand Up @@ -100,7 +118,7 @@ otherwise edited.
### Test

```
npm test
yarn test
```

### Organization
Expand All @@ -116,4 +134,3 @@ its own `package.json` and dependencies.
### License

CC0 for all code unique to this API.

44 changes: 20 additions & 24 deletions api-tests/app.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,18 @@ var settings = require("./settings");
//import { app } from "../../server";
var request = require("supertest");

import * as fs from 'fs';
import * as path from 'path';

import * as fs from "fs";
import * as path from "path";

const endpointsToFix = [
//can't find the gmna.lookup_units table. 'geologic_units_gmna.ts',
//'geologic_units_gmus.ts'
//'mancos_test_cases.ts',
//uses gmus.lookup_units table. queries need to be customized and changed to match all of the lookup tables within postgresql
//can't find the gmna.lookup_units table. 'geologic_units_gmna.ts',
//'geologic_units_gmus.ts'
//'mancos_test_cases.ts',
//uses gmus.lookup_units table. queries need to be customized and changed to match all of the lookup tables within postgresql
//'tiles.ts'
]
];
const testFiles = [
/*'carto_small.ts',
/*'carto_small.ts',
'columns.ts',
'defs.ts',
'defs_columns.ts',
Expand All @@ -42,22 +41,19 @@ const testFiles = [
'mobile_fossil_collections.ts',
'mobile_macro_summary.ts',
'mobile_map_query.ts',*/
'mobile_point.ts',
'mobile_point_details.ts',
'paleogeography.ts',
'root.ts',
'sections.ts',
'stats.ts' ]


"mobile_point.ts",
"mobile_point_details.ts",
"paleogeography.ts",
"root.ts",
"sections.ts",
"stats.ts",
];

const testDir = path.join(__dirname, 'v2Tests');
const testDir = path.join(__dirname, "v2Tests");

testFiles.forEach((file) => {
describe(file, function () {
console.log(`Loading test file: ${file}`);
require(path.join(testDir, file));
});
describe(file, function () {
console.log(`Loading test file: ${file}`);
require(path.join(testDir, file));
});
});


2 changes: 1 addition & 1 deletion api-tests/settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
module.exports = {
host_prod: "https://www.macrostrat.org/api/v2",
host: "http://localhost:5000/v2",
host_dev: "https://dev2.macrostrat.org/api/v2"
host_dev: "https://dev2.macrostrat.org/api/v2",
};
4 changes: 2 additions & 2 deletions api-tests/tsconfig.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
"module": "commonjs",
"strict": true,
"esModuleInterop": true,
"types": ["node", "mocha"], // Add "mocha" here so TypeScript recognizes the global functions
"types": ["node", "mocha"], // Add "mocha" here so TypeScript recognizes the global functions
"skipLibCheck": true
},
"include": ["./v2Tests/**/*.ts"], // Ensure your test files are included
"include": ["./v2Tests/**/*.ts"], // Ensure your test files are included
"exclude": ["node_modules"]
}
58 changes: 29 additions & 29 deletions api-tests/v2Tests/carto_small.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,41 +1,41 @@
var request = require("supertest"),
validators = require("../validators"),
settings = require("../settings");
validators = require("../validators"),
settings = require("../settings");

it("should return metadata", function (done) {
request(settings.host)
.get("/carto/small")
.expect(validators.aSuccessfulRequest)
.expect(validators.json)
.expect(validators.metadata)
.end(function (error: any, res: any) {
if (error) return done(error);
done();
});
request(settings.host)
.get("/carto/small")
.expect(validators.aSuccessfulRequest)
.expect(validators.json)
.expect(validators.metadata)
.end(function (error: any, res: any) {
if (error) return done(error);
done();
});
});

it("should return a sample", function (done) {
request(settings.host)
.get("/carto/small?sample")
.expect(validators.aSuccessfulRequest)
.expect(validators.json)
.expect(validators.aSample)
.end(function (error: any, res: any) {
if (error) return done(error);
done();
});
request(settings.host)
.get("/carto/small?sample")
.expect(validators.aSuccessfulRequest)
.expect(validators.json)
.expect(validators.aSample)
.end(function (error: any, res: any) {
if (error) return done(error);
done();
});
});

it("should accept a latitude and longitude", function (done) {
request(settings.host)
.get("/carto/small?lat=43&lng=-89.3")
.expect(validators.aSuccessfulRequest)
.expect(validators.json)
.expect(validators.atLeastOneResult)
.end(function (error: any, res: any) {
if (error) return done(error);
done();
});
request(settings.host)
.get("/carto/small?lat=43&lng=-89.3")
.expect(validators.aSuccessfulRequest)
.expect(validators.json)
.expect(validators.atLeastOneResult)
.end(function (error: any, res: any) {
if (error) return done(error);
done();
});
});

/*
Expand Down
Loading