docker: pre-download all hub items and data, opt-in hub update/upgrade (#2933)

* docker: pre-download all hub items and data, opt-in hub update/upgrade

* docker/bars: don't purge anything before pre-downloading hub

* Docker: README update

Author: mmetc

Dockerfile

@@ -25,6 +25,7 @@ RUN make clean release DOCKER_BUILD=1 BUILD_STATIC=1 && \
     ./wizard.sh --docker-mode && \
     cd - >/dev/null && \
     cscli hub update && \
+    ./docker/preload-hub-items && \
     cscli collections install crowdsecurity/linux && \
     cscli parsers install crowdsecurity/whitelists
 

docker/README.md

@@ -134,7 +134,6 @@ labels:
   type: apache2
 ```
 
-
 ## Recommended configuration
 
 ### Volumes
@@ -146,6 +145,14 @@ to avoid losing credentials and decision data in case of container destruction a
 * Acquisition: `/etc/crowdsec/acquis.d` and/or `/etc/crowdsec.acquis.yaml` (yes, they can be nested in `/etc/crowdsec`)
 * Database when using SQLite (default): `/var/lib/crowdsec/data`
 
+### Hub updates
+
+To ensure you have the latest version of the collections, scenarios, parsers, etc., you can set the variable `DO_HUB_UPGRADE` to true.
+This will perform an update/upgrade of the hub every time the container is started.
+
+Be aware that if your container is misbehaving and caught in a restart loop, the CrowdSec hub may ban your IP for some time and your containers
+will run with the version of the hub that is cached in the container's image. If you enable `DO_HUB_UPGRADE`, do it when your infrastructure is running
+correctly and make sure you have some monitoring in place.
 
 ## Start a Crowdsec instance
 
@@ -316,7 +323,7 @@ config.yaml) each time the container is run.
 | `BOUNCERS_ALLOWED_OU`   | bouncer-ou | OU values allowed for bouncers, separated by comma |
 |                         | | |
 | __Hub management__      | | |
-| `NO_HUB_UPGRADE`        | false | Skip hub update / upgrade when the container starts |
+| `DO_HUB_UPGRADE`        | false | Force hub update / upgrade when the container starts. If for some reason the container restarts too often, it may lead to a temporary ban from hub updates. |
 | `COLLECTIONS`           | | Collections to install, separated by space: `-e COLLECTIONS="crowdsecurity/linux crowdsecurity/apache2"` |
 | `PARSERS`               | | Parsers to install, separated by space |
 | `SCENARIOS`             | | Scenarios to install, separated by space |

docker/docker_start.sh

@@ -304,9 +304,8 @@ conf_set_if "$PLUGIN_DIR" '.config_paths.plugin_dir = strenv(PLUGIN_DIR)'
 
 ## Install hub items
 
-cscli hub update || true
-
-if isfalse "$NO_HUB_UPGRADE"; then
+if istrue "$DO_HUB_UPGRADE"; then
+    cscli hub update || true
     cscli hub upgrade || true
 fi
 

docker/preload-hub-items

@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+
+set -eu
+
+# pre-download everything but don't install anything
+
+echo "Pre-downloading Hub content..."
+
+types=$(cscli hub types -o raw)
+
+for itemtype in $types; do
+    ALL_ITEMS=$(cscli "$itemtype" list -a -o json | itemtype="$itemtype" yq '.[env(itemtype)][] | .name')
+    if [[ -n "${ALL_ITEMS}" ]]; then
+        #shellcheck disable=SC2086
+        cscli "$itemtype" install \
+            $ALL_ITEMS \
+            --download-only \
+            --error
+    fi
+done
+
+echo " done."

test/bin/preload-hub-items

@@ -9,20 +9,12 @@ THIS_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
 
 # pre-download everything but don't install anything
 
-echo -n "Purging existing hub..."
+echo "Pre-downloading Hub content..."
 
 types=$("$CSCLI" hub types -o raw)
 
 for itemtype in $types; do
-    "$CSCLI" "${itemtype}" delete --all --error --purge --force
-done
-
-echo " done."
-
-echo -n "Pre-downloading Hub content..."
-
-for itemtype in $types; do
-    ALL_ITEMS=$("$CSCLI" "$itemtype" list -a -o json | jq --arg itemtype "$itemtype" -r '.[$itemtype][].name')
+    ALL_ITEMS=$("$CSCLI" "$itemtype" list -a -o json | itemtype="$itemtype" yq '.[env(itemtype)][] | .name')
     if [[ -n "${ALL_ITEMS}" ]]; then
         #shellcheck disable=SC2086
         "$CSCLI" "$itemtype" install \
@@ -32,11 +24,4 @@ for itemtype in $types; do
     fi
 done
 
-# XXX: download-only works only for collections, not for parsers, scenarios, postoverflows.
-# so we have to delete the links manually, and leave the downloaded files in place
-
-for itemtype in $types; do
-    "$CSCLI" "$itemtype" delete --all --error
-done
-
 echo " done."