Add proxy auth support and status/metrics commands

CHANGES.md

@@ -10,6 +10,31 @@
 
 ## Version 1.3.0
 
+- ➕ Added authenticated proxy support. Proxy settings can be provided through
+  environment variables (`CVDUPDATE_PROXY_URL`, `CVDUPDATE_PROXY_USER`,
+  `CVDUPDATE_PROXY_PASS`) or with `cvd config set` (`--proxy-url`,
+  `--proxy-user`, `--proxy-pass`). Credentials are embedded in the proxy URL so
+  the `Proxy-Authorization` header is sent, and they are redacted from logs and
+  from `cvd config show`. Feature courtesy of Nik Kale.
+
+  Closes #7, #9.
+
+  [GitHub Pull-Request](https://github.com/Cisco-Talos/cvdupdate/pull/81)
+
+- ➕ Added a `cvd health` command that reports the health and currency of
+  downloaded databases, including per-database version comparison, file age,
+  and cooldown state. Pass `--json` for machine-readable output and `--check`
+  for a non-zero exit code when databases are not healthy, for scripted health
+  checks. Feature courtesy of Nik Kale.
+
+  [GitHub Pull-Request](https://github.com/Cisco-Talos/cvdupdate/pull/81)
+
+- ➕ Added a `cvd metrics` command that outputs database status as
+  Prometheus-format metrics, either once to stdout or from a persistent HTTP
+  server with `--serve` for scraping. Feature courtesy of Nik Kale.
+
+  [GitHub Pull-Request](https://github.com/Cisco-Talos/cvdupdate/pull/81)
+
 - ➕ Added a `cvd status` command (alias `s`) that reports the status of all
   databases, or of a single database when given a name. Pass `--json` for
   machine-readable output. The `cvd list` command (alias `ls`) now prints just

README.md

@@ -184,9 +184,45 @@ export https_proxy
 cvd update -V
 ```
 
-> _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/<version> (<uuid>)` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/state.json` file (or `~/.cvdupdate/config.json` for cvdupdate <=1.0.2). See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details.
->
-> Adding support for proxy authentication is a ripe opportunity for a community contribution to the project.
+### Using an authenticated proxy
+
+For proxies that require authentication, CVD-Update accepts proxy credentials
+through environment variables or the config file. Credentials are embedded in
+the proxy URL so the `Proxy-Authorization` header is sent, and they are redacted
+from logs and from `cvd config show`.
+
+Using environment variables:
+
+```bash
+CVDUPDATE_PROXY_URL="http://proxy.example.com:8080" \
+CVDUPDATE_PROXY_USER="myuser" \
+CVDUPDATE_PROXY_PASS="mypassword" \
+cvd update -V
+```
+
+Or store them in the config file:
+
+```bash
+cvd config set --proxy-url http://proxy.example.com:8080
+cvd config set --proxy-user myuser
+cvd config set --proxy-pass         # prompts for the password
+```
+
+Environment variables take precedence over config file settings, and special
+characters in credentials are URL-encoded automatically.
+
+> _Security note_: credentials passed with `--proxy-user`/`--proxy-pass` are
+> stored **in plaintext** in `config.json` (the file is created with `0600`
+> permissions, i.e. readable only by your user). If you would rather not persist
+> the password to disk, provide the `CVDUPDATE_PROXY_*` environment variables at
+> runtime instead. Avoid `cvd update -D` (debug mode) when a proxy is
+> configured: it prints raw HTTP headers, including the `Proxy-Authorization`
+> credentials, to stdout.
+
+> _Note_: the proxy only carries CVD-Update's HTTP traffic. CVD-Update still
+> resolves database versions over DNS first, so the DNS server must remain
+> reachable (see [Using a proxy](#using-a-proxy) for the `--nameservers`
+> workaround on networks that block outbound DNS).
 
 ## Files and directories created by CVD-Update
 
@@ -235,6 +271,27 @@ cvd status daily.cvd
 
 > _Note_: `status` replaces the old `show` command. `show` still works as a deprecated alias and will be removed in a future release. Add `--json` to `status` (and `list`) for machine-readable output.
 
+Check the health and currency of downloaded databases. This compares the local
+version against the version advertised over DNS and reports each database as
+current, behind, unknown, or missing, along with file age and cooldown state.
+
+```bash
+cvd health
+cvd health --json           # machine-readable output
+cvd health --check          # non-zero exit code when not healthy (for scripts)
+```
+
+The `--check` exit codes are `0` (healthy), `1` (warning), and `2` (critical),
+which makes `cvd health --check` convenient for cron jobs and monitoring hooks.
+
+Export database status as Prometheus metrics, either once to stdout or from a
+persistent HTTP server for scraping.
+
+```bash
+cvd metrics                 # print metrics once to stdout
+cvd metrics --serve         # serve metrics at http://127.0.0.1:9090/metrics
+```
+
 Print out the config to see what it looks like.
 
 ```bash
@@ -278,6 +335,9 @@ cvd config show --json         # current configuration
 | `--dbs-directory`, `-d` | Database directory path (your HTTP server's `www` root). |
 | `--cdiffs-rotate` / `--no-cdiffs-rotate` | Rotate (delete) old CDIFF files (default: on). |
 | `--cdiffs-to-keep` | Number of CDIFFs to keep per database when rotating (default `30`). |
+| `--proxy-url` | Proxy URL, e.g. `http://proxy.example.com:8080` (see [Using an authenticated proxy](#using-an-authenticated-proxy)). |
+| `--proxy-user` | Proxy username. |
+| `--proxy-pass` | Proxy password; pass the flag with no value to be prompted. Stored in plaintext (`config.json`, mode `0600`). |
 | `--state-file` | Path to the state file (stores versions, UUID, and per-database metadata). |
 
 ```bash

cvdupdate/__main__.py

@@ -301,13 +301,19 @@ def config():
               help="Number of CDIFF files to keep.")
 @click.option("--state-file", type=click.Path(), default="",
               help="Path to the state file.")
+@click.option("--proxy-url", type=str, default="",
+              help="Proxy URL (e.g. http://proxy.example.com:8080).")
+@click.option("--proxy-user", type=str, default="",
+              help="Proxy username.")
+@click.option("--proxy-pass", type=str, default="", is_flag=False, flag_value="__PROMPT__",
+              help="Proxy password. Use the flag without a value to be prompted.")
 # Deprecated flag names from <= 1.2.0, kept as hidden aliases for backward compatibility.
 @click.option("--nameserver", type=str, default="", hidden=True)
 @click.option("--logdir", type=click.Path(), default="", hidden=True)
 @click.option("--dbdir", type=click.Path(), default="", hidden=True)
 def config_set(ctx, config, verbose, nameservers, max_retries, logs_enabled, logs_directory,
                logs_rotate, logs_to_keep, dbs_directory, cdiffs_rotate, cdiffs_to_keep,
-               state_file, nameserver, logdir, dbdir):
+               state_file, proxy_url, proxy_user, proxy_pass, nameserver, logdir, dbdir):
     """
     Set configuration options.
 
@@ -350,10 +356,18 @@ def config_set(ctx, config, verbose, nameservers, max_retries, logs_enabled, log
         and cdiffs_rotate is None
         and cdiffs_to_keep == 0
         and state_file == ""
+        and proxy_url == ""
+        and proxy_user == ""
+        and proxy_pass == ""
     )
     if no_options_set:
         click.echo(ctx.get_help())
         return
+
+    # Prompt for the proxy password when the flag was given without a value.
+    if proxy_pass == "__PROMPT__":
+        proxy_pass = click.prompt("Proxy password", hide_input=True)
+
     CVDUpdate(
         config=config,
         verbose=verbose,
@@ -366,6 +380,9 @@ def config_set(ctx, config, verbose, nameservers, max_retries, logs_enabled, log
         dbs_directory=dbs_directory,
         cdiffs_rotate=cdiffs_rotate,
         cdiffs_to_keep=cdiffs_to_keep,
+        proxy_url=proxy_url,
+        proxy_user=proxy_user,
+        proxy_pass=proxy_pass,
         state_file=state_file,
     )
 
@@ -379,10 +396,18 @@ def config_show(config: str, verbose: bool, as_json: bool):
     Print out the current configuration.
     """
     m = CVDUpdate(config=config, verbose=verbose)
+
+    # Redact proxy credentials: mask the password and strip any userinfo in the URL.
+    display = dict(m.config)
+    if display.get('proxy_pass'):
+        display['proxy_pass'] = '********'
+    if display.get('proxy_url'):
+        display['proxy_url'] = m._sanitize_proxy_url(display['proxy_url'])
+
     if as_json:
-        print(_json.dumps(m.config, indent=4))
+        print(_json.dumps(display, indent=4))
     else:
-        for key, value in m.config.items():
+        for key, value in display.items():
             cli_key = key.replace('_', '-')
             if value == "" or value is None:
                 print(f"{cli_key}:")
@@ -428,6 +453,208 @@ def clean_all(config: str, verbose: bool):
     m.clean_all()
 
 
+@cli.command("health")
+@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path.")
+@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output.")
+@click.option("--json", "-j", "output_json", is_flag=True, default=False, help="Output in JSON format.")
+@click.option("--check", is_flag=True, default=False, help="Exit with non-zero status if databases are not healthy.")
+def db_health(config: str, verbose: bool, output_json: bool, check: bool):
+    """
+    Check the health and currency of downloaded databases.
+
+    Reports each database's local and remote version, file age, and version
+    status (current, behind, unknown, or missing). Use --check for scripted
+    health checks that return a non-zero exit code on problems.
+
+    Version status comes from comparing the local version against the version
+    advertised over DNS. When DNS is unavailable the status is reported as
+    unknown rather than behind. The Age column is informational, since some
+    databases such as main.cvd change infrequently and a current mirror can
+    hold an old but correct file. Age thresholds are fixed at 24, 48, and 72
+    hours and are not yet configurable per database.
+    """
+    import datetime
+
+    # Logs to stderr so `health --json` output stays valid JSON.
+    m = CVDUpdate(config=config, verbose=verbose, log_to_stderr=True)
+    status = m.db_status()
+    summary = status['summary']
+
+    if output_json:
+        click.echo(_json.dumps(status, indent=2))
+    else:
+        databases = status['databases']
+        warnings = status['warnings']
+
+        overall = summary['overall_status'].upper()
+        if overall == 'HEALTHY':
+            status_color = Fore.GREEN
+        elif overall == 'WARNING':
+            status_color = Fore.YELLOW
+        else:
+            status_color = Fore.RED
+
+        last_check = datetime.datetime.fromtimestamp(summary['last_check'], tz=datetime.timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC')
+
+        click.echo("")
+        click.echo("CVD-Update Database Health")
+        click.echo("==========================")
+        click.echo(f"Overall Status: {status_color}{overall}{Style.RESET_ALL}")
+        click.echo(f"Last Check: {last_check}")
+        click.echo("")
+
+        click.echo(f"{'Database':<16} {'Local':<8} {'Remote':<8} {'Age':<10} {'Status':<12} {'Size':<10}")
+        click.echo(f"{'-'*16} {'-'*8} {'-'*8} {'-'*10} {'-'*12} {'-'*10}")
+
+        for db in databases:
+            name = db['name']
+            local_ver = str(db['local_version']) if db['local_version'] is not None else '-'
+            remote_ver = str(db['remote_version']) if db['remote_version'] is not None else '-'
+
+            if db['age_hours'] is None:
+                age_str = '-'
+            elif db['age_hours'] < 1:
+                age_str = f"{int(db['age_hours'] * 60)}m"
+            elif db['age_hours'] < 24:
+                age_str = f"{int(db['age_hours'])}h"
+            else:
+                days = int(db['age_hours'] / 24)
+                hours = int(db['age_hours'] % 24)
+                age_str = f"{days}d {hours}h"
+
+            # Color by version state (the Age column already shows file age).
+            if db['is_missing']:
+                status_str = f"{Fore.RED}MISSING{Style.RESET_ALL}"
+            elif db['version_status'] == 'current':
+                status_str = f"{Fore.GREEN}CURRENT{Style.RESET_ALL}"
+            elif db['version_status'] == 'outdated':
+                status_str = f"{Fore.RED}BEHIND{Style.RESET_ALL}"
+            else:
+                status_str = f"{Fore.YELLOW}UNKNOWN{Style.RESET_ALL}"
+
+            if db['file_size_bytes'] is None:
+                size_str = '-'
+            elif db['file_size_bytes'] < 1024:
+                size_str = f"{db['file_size_bytes']} B"
+            elif db['file_size_bytes'] < 1024 * 1024:
+                size_str = f"{db['file_size_bytes'] / 1024:.1f} KB"
+            else:
+                size_str = f"{db['file_size_bytes'] / (1024 * 1024):.1f} MB"
+
+            # The status column needs extra width to account for color codes.
+            click.echo(f"{name:<16} {local_ver:<8} {remote_ver:<8} {age_str:<10} {status_str:<23} {size_str:<10}")
+
+        click.echo("")
+
+        if warnings:
+            click.echo(f"{Fore.YELLOW}Warnings:{Style.RESET_ALL}")
+            for warning in warnings:
+                click.echo(f"  - {warning}")
+            click.echo("")
+
+        click.echo(f"Summary: {summary['current_count']}/{summary['total_databases']} databases current, {len(warnings)} warnings")
+        click.echo("")
+
+    if check:
+        if summary['overall_status'] == 'healthy':
+            sys.exit(0)
+        elif summary['overall_status'] == 'warning':
+            sys.exit(1)
+        else:  # critical
+            sys.exit(2)
+
+
+@cli.command("metrics")
+@click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path.")
+@click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output.")
+@click.option("--serve", "-s", is_flag=True, default=False, help="Start HTTP server for Prometheus scraping.")
+@click.option("--port", "-p", type=int, default=9090, help="Port for metrics server. Default: 9090.")
+@click.option("--bind", "-b", type=str, default="127.0.0.1", help="Address to bind metrics server. Default: 127.0.0.1.")
+@click.option("--cache-ttl", type=int, default=60, help="Seconds to cache status between scrapes in --serve mode. Default: 60.")
+def db_metrics(config: str, verbose: bool, serve: bool, port: int, bind: str, cache_ttl: int):
+    """
+    Output Prometheus metrics for monitoring.
+
+    By default, outputs metrics to stdout for one-shot collection.
+    Use --serve to start a persistent HTTP server for Prometheus scraping.
+    """
+    from cvdupdate.metrics import PrometheusMetrics
+    import http.server
+    import threading
+    import time
+
+    # Logs to stderr so `cvd metrics > file` stays valid Prometheus text.
+    m = CVDUpdate(config=config, verbose=verbose, log_to_stderr=True)
+
+    if serve:
+        # Cache status between scrapes so each request doesn't trigger a live
+        # DNS query.
+        cache_lock = threading.Lock()
+        status_cache = {'status': None, 'time': 0.0}
+
+        def get_cached_status():
+            now = time.time()
+            with cache_lock:
+                if status_cache['status'] is None or (now - status_cache['time']) > cache_ttl:
+                    status_cache['status'] = m.db_status()
+                    status_cache['time'] = now
+                return status_cache['status']
+
+        class MetricsHandler(http.server.BaseHTTPRequestHandler):
+            verbose_mode = verbose
+            timeout = 10  # don't let a slow client stall scrapes
+
+            def do_GET(self):
+                if self.path == '/metrics' or self.path == '/':
+                    try:
+                        status = get_cached_status()
+                        content = PrometheusMetrics(status).generate().encode('utf-8')
+                    except Exception as exc:
+                        # Report 500 rather than dropping the connection.
+                        m.logger.error(f'Failed to generate metrics: {exc}')
+                        self.send_response(500)
+                        self.send_header('Content-Type', 'text/plain; charset=utf-8')
+                        self.end_headers()
+                        self.wfile.write(b'error generating metrics\n')
+                        return
+
+                    self.send_response(200)
+                    self.send_header('Content-Type', 'text/plain; version=0.0.4; charset=utf-8')
+                    self.send_header('Content-Length', str(len(content)))
+                    self.end_headers()
+                    self.wfile.write(content)
+                elif self.path == '/health':
+                    self.send_response(200)
+                    self.send_header('Content-Type', 'text/plain')
+                    self.end_headers()
+                    self.wfile.write(b'OK')
+                else:
+                    self.send_response(404)
+                    self.end_headers()
+
+            def log_message(self, format, *args):
+                if self.verbose_mode:
+                    m.logger.debug(f'{self.address_string()} - {format % args}')
+
+        m.logger.info(f'Starting metrics server on {bind}:{port}')
+        m.logger.info(f'Metrics available at http://{bind}:{port}/metrics')
+
+        # Threaded (a slow client can't block scrapes) with address reuse (a
+        # quick restart won't hit EADDRINUSE).
+        class _MetricsServer(http.server.ThreadingHTTPServer):
+            daemon_threads = True
+
+        with _MetricsServer((bind, port), MetricsHandler) as httpd:
+            try:
+                httpd.serve_forever()
+            except KeyboardInterrupt:
+                m.logger.info('Metrics server stopped')
+    else:
+        status = m.db_status()
+        metrics = PrometheusMetrics(status)
+        click.echo(metrics.generate())
+
+
 @cli.command("serve")
 @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path.")
 @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output.")