dependency updates and export fixes (#195)

* [web-app] test insecure transport resolution

* [web-app] fortify: try to resolve insecure random false positives

* [web-app] fortify: try to resolve insecure transport false positive findings

* [service] delete obsolete openapi doc

* [web-app] set post method on forms to avoid fortify scan password management false positive

* [web-app] set autocomplete false on password fields to resolve fortify scan findings

* [service] fortify scan: rename http variable to resolve insecure transport false positive

* [plugins/nga-msi] owasp dep check: upgrade axios dependency

* [service] owasp dep check: upgrade moment and pug

* [service] owasp dep check: upgrade xmldom

* [service] owasp dep check: remove unused dep @mapbox/togeojson

* [service] owasp dep check: remove unused deps terraformer, terrformer-arcgis-parser

* [plugins/nga-msi] bump patch version

* [service] owasp dep check: upgrade async package

* [service] owasp dep check: upgrade body-parser package to transitively upgrade qs package and sync with express

* [web-app] owasp dep check: upgrade underscore

* [service] owasp dep check: upgrade cpy-cli to remove trim-newlines

* [service] owasp dep check: upgrade multer to remove dicer dep

* [service] fortify scan: remove localhost http server from openapi doc to resolve insecure transport finding

* bump core versions to 6.2.10-beta.0

* [service] update better-sqlite3 to 8.6+ so the build works with node 20 on macos

* [service] owasp dep check: upgrade jsonwebtoken

* [service] owasp dep check: upgrade passport before upgrading related passport plugins

* [service] owasp dep check: removed apparently unused openid-client dependency to resolove finding in transitive decode-uri-component

* [service] owasp dep check: upgrade passport-saml to @node-saml/passport-saml

* [service] owasp dep check: upgrade express

* [service] remove unnecessary type cast

* [service] fix type conflict with @node-saml/passport-saml request user property

* resolve type conflicts between service and image plugin

* [plugins/nga-msi] generate new package-lock, bump peer mage.service beta version, bump patch version

* [plugins/image] bump service patch version

* [service] update xmldom imports to scoped package

* [service] fix timestamp bug in test

* [service] move top level eslint config to service

* [service] update eslint

* [ci] remove node 14 from the testing list

* [service] improve eslint config

* [service] remove unused geojson-vt dependency

* [service] add error logging to kml layer import

* [service] layer routes:
- remove unused pbf format handling which would crash if exercised anyway
- refactor copy-pasted handler for getting geopackage tiles

* [web-app] remove unused leaflet.vectorgrid dependency

* [service] remove unused jose dependency

* [docker] add saml test container

* [web-app] consider port when comparing origin of idp popup window login result message

* [web-app][service] post json for auth configuration instead of form url encoded with json string

* [web-app][service] stop erroneously using old options key in saml settings

* [service][web-app] make saml cert required, because passport-saml now requires cert

* [service][web-app] change saml privateCert setting to privateKey to match passport-saml

* [service] add db migration to move saml settings.options entries to settings

* changelog

* [service] rename export modules to typescript modules

* [service] make base exporter a proper class

* [service] make csv exporter a proper class

* [service] fix type definitions on legacy models

* [service] cleanup types in csv exporter

* [service] fix potential null referenced in csv exporter

* [service] add mage event entity to base exporter

* [service] add types for archiver and json2csv

* [service] add properties to location entity

* [service] change kml exporter to proper class

* [service] change kmlWriter module to functions instead of unnecessary class and single exported instance

* [service] minor lint and formatting fixes

* changelog

* [service] wip: exports typescript migration
[skip ci]

* [service] kml export compiles

* [service] add a few doc comments

* [service] fix color function in kml export

* [service] geopackage export: change requires to imports

* [service] geopackage export: remove unnecessary canvas kit setting

* [service] geopackage export: change to class syntax

* [service] geopackage export: add initial type annotations

* [service] geopackage export: fix more type issues and make code safer from reference errors

* [service] geopackage export: fix remaining type issues and close a cursor on the location query

* [service] todo comment

* [service] geojson export: change require statements to import

* [service] geojson export: change to ecmascript class from prototype

* [service] geopackage export: move some methods that never reference `this` to module functions

* [service] geojson export: add types and simplify observation property mapping

* [service] geojson export: fix remaining type errors

* [service] exports: replace exporter factory with index module and typed api

* [service] exports: change routes to typescript file

* [service] exports routes: change requires to imports

* [service] exports: fix type errors in routes and add model type definitions

* fix bloated package-lock in project root

* [service] fix turf centroid type discrepancies

* [service] exports: update tests to use new factory function

* [service] exports: revert to export factory instance to make old tests pass

* [service] exports: get exporter factory tests passing

* [service] exports: fix geopackage test import

* [service] exports: fix csv test import

* [service] fix style copying

* [service] exports: fix csv tests

* [service] exports: fix geopackage tests

* [service] upgrade supertest dependency

* [service] upgrade supertest and superagent types

* [service] upgrade mongodb-memory-server

* [service] add axios for functional tests

* [service] fn-test: add client and server stack for functional tests

* [service] fn-test: add package path mapping in tsconfig

* [service] fix boolean condition to allow 0 as the server port which prompts the http server to pick a random open port

* [service] do not emit the mage ready event until after the http server starts listening

* [service] fn-test:
* undo ts path mapping which only works for types
* start test stack as child process because of global env parameters

* [service] fix validation message

* [service] fn-test: fix setup request in test client

* [service] fn-test: fix response code expectation

* [service] fn-test: fix mongodb cleanup error `StateError`

* [service] fn-test: update mongodb-memory-server dep

* [service] fn-test: add types and methods for more client requests

* [service] fix deprecated function call

* [service] lint error

* [service] fn-test: wip

* [service] fn-test: wip: client with all operations and partial export fixture setup

* [service] upgrade superagent typedefs

* [service] lints and misc cleanup

* [service] fn-test: remove scratch data before running tests

* [service] fn-test: fix icon uploads

* [service] fn-test: add method for adding event participants

* [service] fn-test: add method to archive forms

* [service] fn-test: export test fixtures

* [service] fn-test: add delete user method and deleted user observation in exports test

* [service] fn-test: exports: geopackage export failing on missing icon

* [service] cleanup in geopackage export

* [service] add types for legacy icon modules

* [service] proper path creation

* [service] improve icon model typedef

* [service] fix geopackage export icon null reference

* [service] fn-test: exports: wait for export method was requesting the route to download the export, not get the export information

* [service] fn-test: fix attachment uploads in client

* [service] better error response for attachment uploads with mismatched meta-data

* [service] fn-test: exports: add several attachments to fixture data

* [service] fn-test: exports: fix form field typos

* [service] fn-test: exports: fix test filter to include attachments

* [service] add comments about export attachment api

* [service] fn-test: exports: add remaining export formats in test

* [service] try increasing timeout on test setup to avoid failures in ci

* [service] fix erroneous user icon path in geopackage export

* [service] fn-test: support user icon uploads

* [service] fn-test: exports: add user icons

* [service] fix observation app test for changed attachment store error response

* [service] try again to set the test timeout for downloading mongodb memory server

* [service] remove unsupported attachment relative links from csv export

* [service] fn-test: client: create and delete devices; update forms; return sign-in result

* [service] fn-test: exports: add test cases for deleted devices and invalid stale form entries

* [service] log all uncaught errors from express handler

* bump core versions to 6.2.10-beta.1

* [service] fn-test: remove unnecessary async modifiers from methods; add methods to manage auth providers; make device id optional for sign-in when device auto-registration is enabled

* [service] fn-test: exports: fix failure messages

* [service] fn-test: exports: add fixture data to reproduce locations without device ids that cause exports to fail

* [service] fix device id null reference in geopackage and csv exports

* bump core version to 6.2.10-beta.2

* [plugins/image] upgrade sharp dependency to resolve vulnerability finding

* [plugins/image] bump service patch version to 1.0.6

* [service] rename saml settings migrations with a distinct numeric prefix

* add doc with notes on oauth setup

* [service] fix test for renamed module

* bump core prerelease versions to 6.2.0-beta.3
[skip ci]

* [docker] add iron bank dockerfile; update docker readme; specify platform in compose file

Author: restjohn

.eslintrc.js

@@ -15,7 +15,7 @@ on:
         value: ${{ jobs.main.outputs.node_versions-pre_latest-json }}
 
 env:
-  MAGE_NODE_VERSIONS: 14 16 18
+  MAGE_NODE_VERSIONS: 16 18 20
 
 jobs:
 

.github/workflows/config.yaml

@@ -11,12 +11,22 @@ MAGE adheres to [Semantic Versioning](http://semver.org/).
 
 ## [6.2.10](https://github.com/ngageoint/mage-server/releases/tag/6.2.10)
 
+#### Database Migrations
+* [SAML settings](./service/src/migrations/030-saml-settings.js)
+  This migration modifies documents in the `authenticationconfigurations` collection
+  by moving all entries from `settings.options` into `settings` and removing
+  the `options` entry.  See the related bug fix below.
+
 #### Features
+* Layer file download - The _Layer_ admin web page now includes a button to
+  download layer files when applicable.  This currently works only for
+  GeoPackage layers.
 
-- **Download Layer File Button**:
-  - Added a new "Download" layer file button to the interface.
-  - Allows users to download layer files associated with a specific layer.
-  - Enhanced user experience by enabling easy file downloads.
+#### Bug Fixes
+* The web app and service incorrectly used a `settings.options` sub-document on
+  SAML configurations to read and write some entries that should be in the
+  `settings` entry.
+* Many third-party dependencies were quite outdated or superfluous.
 
 ## [6.2.9](https://github.com/ngageoint/mage-server/releases/tag/6.2.9)
 

CHANGELOG.md

@@ -4,29 +4,22 @@
 
 The MAGE Server image contains the the core MAGE server Node app that consists
 of the ReST web service and the MAGE web app.  By default, the image also
-includes plugins maintained in the [MAGE server repository](../plugins/).  To
-build the image, you will need the NPM package tarballs for the MAGE server
-packages - `@ngageoint/mage.service`, `@ngageoint/mage.web-app`, and any
-available plugin packages.  You can obtain the tarballs by building the
-packages locally and using the `npm pack` command in each package directory,
-download the tarballs from a GitHub [actions workflow run](https://github.com/ngageoint/mage-server/actions),
-or from a GitHub [release](https://github.com/ngageoint/mage-server/releases).
-The server [Dockerfile](server/Dockerfile) expects the tarballs to be
-siblings of the Dockerfile, so move them into the `server` directory.  Then,
-starting the top level of the MAGE server repository:
+includes plugins maintained in the [MAGE server repository](../plugins/).  By
+default, the server [Dockerfile](./server/Dockerfile) pulls the latest tagged
+versions from the NPM registry.  You can override the version using Docker's
+`--build-arg` CLI switch to set the package versions you want in the image.
+If you are building on Apple Silicon hardware, use `--platform linux/amd64` so
+the built image platform matches the base image platform.  Here's an example of
+building the image with an explicit service version.
 ```bash
 $ cd ./docker/server
-$ docker build -t mage-server:<version>
+$ docker build --platform linux/amd64 --build-arg service_version=6.2.10 -t mage-server:<version> .
 ```
-If you are building from a release and have [GitHub CLI](https://cli.github.com/)
-installed, you can also use the [`build_release.sh`](server/build_release.sh)
-utility script.
-```bash
-$ cd ./docker/server
-$ ./build_release.sh 6.2.0
-```
-The script assumes that you have an [authenticated](https://cli.github.com/manual/gh_auth_login)
-GitHub CLI session.
+
+The Iron Bank [Dockerfile](./server/Dockerfile.ironbank) uses a different,
+hardened [base image](https://ironbank.dso.mil/repomap/details;registry1Path=opensource%252Fnodejs%252Fdebian%252Fnodejs)
+from  US DoD's [Iron Bank](https://ironbank.dso.mil/about) repository.  The
+Dockerfile builds exactly the same as the standard Dockerfile.
 
 ## Docker Compose
 

docker/README.md

@@ -0,0 +1,16 @@
+services:
+  mage-idp-saml:
+    container_name: mage-idp-saml
+    image: kristophjunge/test-saml-idp
+    environment:
+      - "SIMPLESAMLPHP_SP_ENTITY_ID=http://localhost:4242"
+      - "SIMPLESAMLPHP_SP_ASSERTION_CONSUMER_SERVICE=http://localhost:4242/auth/saml/callback"
+      - "SIMPLESAMLPHP_SP_SINGLE_LOGOUT_SERVICE=http://localhost:8080/simplesaml/module.php/saml/sp/saml2-logout.php/test-sp"
+    ports:
+      - 8080:8080
+      - 8443:8443
+    volumes:
+      - type: bind
+        source: ./saml/authsources.php
+        target: /var/www/simplesamlphp/config/authsources.php
+

docker/auth-idp/docker-compose.yml

@@ -0,0 +1,23 @@
+<?php
+
+$config = array(
+
+    'admin' => array(
+        'core:AdminPassword',
+    ),
+
+    'example-userpass' => array(
+        'exampleauth:UserPass',
+        'saml.user1:user1pass' => array(
+            'uid' => array('1'),
+            'eduPersonAffiliation' => array('group1'),
+            'email' => '[email protected]',
+        ),
+        'saml.user2:user2pass' => array(
+            'uid' => array('2'),
+            'eduPersonAffiliation' => array('group2'),
+            'email' => '[email protected]',
+        ),
+    ),
+
+);

docker/auth-idp/saml/authsources.php

@@ -22,6 +22,7 @@ services:
       # dockerfile: Dockerfile-debug
       args:
         MAGE_VERSION: "${MAGE_VERSION:-6.2.0}"
+    platform: linux/amd64
     volumes:
       - ./server/resources:/var/lib/mage
     # Comment the ports block to disallow connections directly to the node

docker/docker-compose.yml

@@ -7,13 +7,9 @@ ARG msi_version=latest
 
 LABEL author="NGA"
 
-WORKDIR /root/building
+USER root
 
-RUN apt-get update && apt-get -y install \
-    curl \
-    git \
-    sudo \
-    unzip
+RUN apt-get update
 
 RUN groupadd -r mage \
     && useradd -m -r -s /bin/bash -g mage mage \

docker/server/Dockerfile

@@ -0,0 +1,32 @@
+FROM registry1.dso.mil/ironbank/opensource/nodejs/debian/nodejs:18.19.1
+
+ARG service_version=latest
+ARG web_version=${service_version}
+ARG image_version=latest
+ARG msi_version=latest
+
+LABEL author="NGA"
+
+USER root
+
+RUN apt-get update
+
+RUN groupadd -r mage \
+    && useradd -m -r -s /bin/bash -g mage mage \
+    && mkdir -p /var/lib/mage \
+    && chown mage:mage /var/lib/mage
+
+USER mage
+ENV MAGE_HOME /home/mage/instance
+WORKDIR ${MAGE_HOME}
+RUN ls -l \
+    && npm i --omit dev @ngageoint/mage.service@${service_version} \
+    && npm i --omit dev @ngageoint/mage.web-app@${web_version} \
+    && npm i --omit dev @ngageoint/mage.image.service@${image_version} \
+    && npm i --omit dev @ngageoint/mage.nga-msi@${msi_version} \
+    && ln -s ./node_modules/.bin/mage.service
+
+VOLUME /var/lib/mage
+EXPOSE 4242
+
+ENTRYPOINT [ "./mage.service", "--plugin", "@ngageoint/mage.image.service", "--plugin", "@ngageoint/mage.nga-msi" ]