Merge pull request #1441 from makeabilitylab/1440-logs-url-docs

Stop documenting the web /logs/ URL — it no longer exists (#1440)

Author: jonfroehlich

CLAUDE.md

@@ -51,7 +51,7 @@ A superuser is required to use `/admin` and add content; create one with `python
 - **Push to `master`** → auto-deploys to `makeabilitylab-test.cs.washington.edu` via webhook.
 - **Push a SemVer tag (e.g. `git tag 2.3.2 && git push --tags`)** → deploys to production `makeabilitylab.cs.washington.edu`.
 - Bump `ML_WEBSITE_VERSION` and `ML_WEBSITE_VERSION_DESCRIPTION` in `makeabilitylab/settings.py` when cutting a release.
-- Build logs: `<host>/logs/buildlog.txt`. Application logs: `<host>/logs/debug.log`. See `docs/DEPLOYMENT.md` for SSH paths on `recycle.cs.washington.edu`.
+- Application logs: read `debug.log` over SSH on `makelab1`/`makelab2`/`recycle` under `/cse/web/research/makelab/www[-test]/`. Build logs only reach you via the deploy email. **The web `/logs/` URL is gone — every path under it 404s on both hosts.** Confirm what a server is running with `/version.json` (`git_sha`, not `built_at`). See `docs/DEPLOYMENT.md`.
 
 ### Server access model (important — shapes how anything ships to prod/test)
 
@@ -130,7 +130,7 @@ the existing viewset/serializer pattern and keep `v1` fields additive-only
 - **Prod/test `config.ini` has only a `[Django]` section — no `[Postgres]` section.** Per `settings.py`, a missing `[Postgres]` section means Django uses the fallback `DATABASES` default (`HOST='db'`) — i.e. the dockerized `db` service of the active compose file. A `[Postgres]` section, if added, would override it. So the DB is the in-stack `db` container in **every** environment (no external Postgres); on the servers that's the `db` service in `docker-compose.yml`.
 - `DEBUG` resolution order: `DJANGO_ENV=PROD` forces False → `config.ini [Django] DEBUG` → `DJANGO_ENV=DEBUG` forces True → default False.
 - `TIME_ZONE = 'America/Los_Angeles'`. `ML_WEBSITE_VERSION` in settings is shown in the admin header and used in release tagging.
-- **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `<BASE_DIR>/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — the web-served `/logs/debug.log` depends on that. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard.
+- **Logging (#1283):** `debug.log` lives at `LOG_DIR/debug.log`, where `LOG_DIR` is `$ML_LOG_DIR` or `<BASE_DIR>/media` (`/code/media` in the container). Keep it inside `MEDIA_ROOT` — that's the tree bind-mounted to the shared CSE filesystem, so it's what makes the log readable over SSH at all. `ML_LOG_DIR` is unset everywhere today; it exists for non-`/code` hosts. `MEDIA_ROOT` is web-served, so never log anything sensitive. If the dir isn't writable the file handler degrades to a `NullHandler` rather than crashing `django.setup()`, and since there's no console on the servers that state surfaces via `/version.json` (`log_to_file`) and a superuser-only callout on the admin dashboard.
 
 ### Container startup side effects (`docker-entrypoint.sh`)
 

docs/DEPLOYMENT.md

@@ -75,10 +75,17 @@ git push --tags
 
 ### Verifying Deployment
 
-Check the build log to confirm deployment succeeded:
+Confirm the deployment succeeded with `/version.json`, which reports the running
+build (the `/logs/buildlog.txt` URL this section used to point at now 404s — see
+"Log Files" below):
 
-- **Test:** https://makeabilitylab-test.cs.washington.edu/logs/buildlog.txt
-- **Production:** https://makeabilitylab.cs.washington.edu/logs/buildlog.txt
+```bash
+curl -s https://makeabilitylab-test.cs.washington.edu/version.json | python3 -m json.tool
+```
+
+Match `git_sha` against the commit you pushed — **not `built_at`**, which has read
+fresh while the host served stale code. The build log itself is only available in
+the deploy email sent to maintainers on every push.
 
 ## Versioning
 
@@ -112,7 +119,7 @@ View current and past versions on the [Releases page](https://github.com/makeabi
    git push --tags
    ```
 
-5. Verify deployment via the [production build log](https://makeabilitylab.cs.washington.edu/logs/buildlog.txt)
+5. Verify the deploy landed via [`/version.json`](https://makeabilitylab.cs.washington.edu/version.json) — check `git_sha`, not `built_at`
 
 ## Server Configuration
 
@@ -175,18 +182,24 @@ You only do this once per property (not per content change):
 
 ### Log Files
 
-Not all logs live in the same place. Only the Django application log is on
-the shared CSE filesystem (and therefore readable from `recycle`); the build
-and web-server logs live on the Docker host (`grabthar` / `docker-test2`),
-which we can't SSH into — reach those via the web `/logs/` URL or the deploy
-email.
+Not all logs live in the same place. Only the Django application log is on the
+shared CSE filesystem, and it is the only one we can still read directly. The
+build and web-server logs live on the Docker host (`grabthar` / `docker-test2`),
+which we can't SSH into.
+
+> **The web `/logs/` URL is gone.** It used to expose these files over HTTP.
+> Every path under it now 404s on both prod and test (verified 2026-07-28): the
+> response comes back as `Server: gunicorn` with Django's custom 404 template,
+> meaning Apache has no `/logs/` alias any more and the request falls through to
+> Django. It is not coming back — web access to logs is no longer needed and UW
+> CSE IT has trouble maintaining it. Use SSH.
 
 | Log | Description | Where to find it |
 |-----|-------------|------------------|
-| `debug.log` | Django application logs | **On the shared filesystem** — read via SSH on `recycle` (see below) or the web `/logs/` URL. A rotated `debug.log.1` sits alongside it. |
-| `buildlog.txt` | Deployment build output | **Not on the shared filesystem** (so *not* under `www/` on `recycle`). It lives on the Docker host and is emailed to maintainers on every push — that email is the most reliable copy. Also exposed at the web `/logs/` URL. |
-| `httpd-access.log` | HTTP request logs | On the Docker host — web `/logs/` URL. |
-| `httpd-error.log` | HTTP error logs | On the Docker host — web `/logs/` URL. |
+| `debug.log` | Django application logs | **On the shared filesystem** — read via SSH (see below). Rotated `debug.log.1` … `.6` sit alongside it. |
+| `buildlog.txt` | Deployment build output | **Not on the shared filesystem** (so *not* under `www/`). It lives on the Docker host and is emailed to maintainers on every push — **that email is the only copy you can get.** |
+| `httpd-access.log` | HTTP request logs | On the Docker host — **not reachable**; ask UW CSE IT if you need it. |
+| `httpd-error.log` | HTTP error logs | On the Docker host — **not reachable**; ask UW CSE IT if you need it. |
 
 #### Where `debug.log` is written, and what happens if that fails (#1283)
 
@@ -201,15 +214,16 @@ LOG_FILE = $LOG_DIR/debug.log                            # /code/media/debug.log
 out to the shared CSE filesystem — it's what makes `debug.log` readable over SSH at
 `/cse/web/research/makelab/www/debug.log` (prod) and `www-test/debug.log` (test).
 
-> **Note:** `https://<host>/logs/debug.log` **404s on both prod and test** as of
-> 2026-07-28 (verified with `curl`; it falls through to Django's custom 404). The
-> web-URL rows in the table above are stale. SSH is the reliable path — see
-> "Reading `debug.log` over SSH" below. Because the log nonetheless lives in a
-> web-served tree, we still log at INFO rather than DEBUG when `DEBUG` is off.
+Note that `MEDIA_ROOT` is a **web-served** tree, so keep the log conservative:
+we log at INFO rather than DEBUG whenever `DEBUG` is off, and nothing personal or
+sensitive should ever be written to it.
 
 - **`ML_LOG_DIR`** is an optional environment override for hosts that don't use
   `/code`. It is **not set** on prod, test, or local dev, and shouldn't need to
-  be. If you do set it outside `MEDIA_ROOT`, the web `/logs/` URL stops working.
+  be. If you point it outside `MEDIA_ROOT`, the log stops being bind-mounted to
+  the shared filesystem and you lose SSH access to it — which, with no shell on
+  these servers, means losing all access. Only do that if you also arrange a
+  mount for the new location.
 - **If the log directory can't be created or written**, Django does *not* crash
   (it used to: `LOGGING` is evaluated at `django.setup()`, so a bad path killed
   startup before a single request). The file handler degrades to a `NullHandler`
@@ -234,8 +248,8 @@ curl -s $HOST/version.json | python3 -m json.tool
 ```
 
 Match `git_sha` — **not `built_at`**, which has shown fresh on a stuck auto-deploy
-serving stale code. Then confirm records are really being written (the web `/logs/`
-URL 404s, so this has to be SSH):
+serving stale code. Then confirm records are really being written (this has to be
+SSH; there is no web path to the log):
 
 ```bash
 ssh makelab1                                          # or makelab2 / recycle
@@ -244,21 +258,20 @@ tail -5 /cse/web/research/makelab/www-test/debug.log   # timestamps after the de
 ```
 
 The log rotates at 5 MB, so check `debug.log.1` too when hunting a container-start
-sequence.
-
-### Accessing Logs via Web
+sequence. Rotation is currently unreliable under multiple Gunicorn workers — see
+issue #1439 — so a rotated file may be much smaller than 5 MB and may be missing
+records.
 
-- **Test:** https://makeabilitylab-test.cs.washington.edu/logs/
-- **Production:** https://makeabilitylab.cs.washington.edu/logs/
+### Reading `debug.log` over SSH
 
-Only `debug.log` (the Django application log) is reachable this way — it is
-the one log mounted out to the shared CSE filesystem. `buildlog.txt` and the
-`httpd-*.log` files are **not** here (see the table above).
+This is the only way to read the application log. SSH access is read-mostly: you
+can read the shared CSE filesystem, but there is no Docker or `manage.py` access
+on the hosts that run the stack.
 
-1. SSH into the jump host:
+1. SSH to a host with the shared filesystem mounted:
 
    ```bash
-   ssh recycle.cs.washington.edu
+   ssh makelab1     # or makelab2, or recycle.cs.washington.edu
    ```
 
 2. Navigate to the log directory:

makeabilitylab/settings.py

@@ -175,8 +175,10 @@ def _file_log_handler(log_file, level, enabled):
 
 
 # NOTE: this default must stay in sync with MEDIA_ROOT (defined further down as
-# os.path.join(BASE_DIR, 'media')) — the web-served /logs/debug.log URL only works
-# because the log lives inside the media root. MEDIA_ROOT isn't defined yet here
+# os.path.join(BASE_DIR, 'media')) — media/ is the tree bind-mounted out to the
+# shared CSE filesystem, so keeping the log inside it is what makes debug.log
+# readable over SSH at all. There is no shell on these hosts, so a log written
+# anywhere else is a log nobody can read. MEDIA_ROOT isn't defined yet here
 # (LOGGING has to be built before it), hence the duplicated expression;
 # test_default_log_file_is_under_media_root pins the two together.
 LOG_DIR = os.environ.get('ML_LOG_DIR', os.path.join(BASE_DIR, 'media'))
@@ -217,13 +219,12 @@ def _file_log_handler(log_file, level, enabled):
     'handlers': {
         # The file handler writes LOG_FILE (media/debug.log by default), which lands
         # in the bind-mounted web root — that's what makes it readable over SSH at
-        # /cse/web/research/makelab/www[-test]/debug.log. (docs/DEPLOYMENT.md also
-        # describes a /logs/ URL per Jason Howe's design, but that URL 404s on both
-        # prod and test as of 2026-07-28.) Since the file still sits in a web-served
-        # tree, stay conservative: log at INFO when DEBUG is off, but keep DEBUG-level
-        # file logging in local dev where DEBUG is on and nothing is public. If the
-        # log dir isn't writable (LOG_TO_FILE is False), degrade to a NullHandler so
-        # startup never dies (issue #1283).
+        # /cse/web/research/makelab/www[-test]/debug.log, the only way to read it now
+        # that the /logs/ URL is gone. media/ IS a web-served tree, so stay
+        # conservative about what lands here: log at INFO when DEBUG is off, but keep
+        # DEBUG-level file logging in local dev where DEBUG is on and nothing is
+        # public. If the log dir isn't writable (LOG_TO_FILE is False), degrade to a
+        # NullHandler so startup never dies (issue #1283).
         'file': _file_log_handler(LOG_FILE, 'DEBUG' if DEBUG else 'INFO', LOG_TO_FILE),
         'console': {
             'level': 'DEBUG',
@@ -453,8 +454,9 @@ def _file_log_handler(log_file, level, enabled):
 # See: https://docs.djangoproject.com/en/4.2/ref/settings/#media-url
 #
 # NOTE: LOG_DIR (defined up with LOGGING, which has to be built before this) hard-codes
-# the same expression, because the web-served /logs/debug.log URL only works while the
-# log file lives inside the media root. If you move MEDIA_ROOT, move LOG_DIR with it —
+# the same expression, because this is the tree bind-mounted to the shared CSE
+# filesystem — the log has to live inside it to be readable over SSH, which is the only
+# access we have. If you move MEDIA_ROOT, move LOG_DIR with it —
 # test_default_log_file_is_under_media_root fails loudly if the two ever diverge.
 MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
 MEDIA_URL = '/media/'

website/tests/test_logging_config.py

@@ -88,13 +88,15 @@ def test_disabled_degrades_to_nullhandler(self):
 
 class LogFileLocationTests(SimpleTestCase):
     def test_default_log_file_is_under_media_root(self):
-        """The log must live inside MEDIA_ROOT or the /logs/ URL breaks.
+        """The log must live inside MEDIA_ROOT or it becomes unreadable.
 
         ``LOG_DIR`` and ``MEDIA_ROOT`` are computed independently in settings.py
         (LOGGING has to be built before MEDIA_ROOT is defined), so this pins them
-        together: the web-served ``/logs/debug.log`` on -test and prod works only
-        because the log file sits inside the bind-mounted media root. Skipped when
-        ``ML_LOG_DIR`` is set, since an explicit override may point elsewhere.
+        together: media/ is the tree bind-mounted to the shared CSE filesystem, and
+        SSH to that filesystem is the *only* way to read debug.log on -test and prod
+        (there is no shell on those hosts, and the old /logs/ URL is gone). A log
+        written outside it is a log nobody can read. Skipped when ``ML_LOG_DIR`` is
+        set, since an explicit override may point elsewhere.
         """
         if os.environ.get("ML_LOG_DIR"):
             self.skipTest("ML_LOG_DIR override in effect")

website/views/version.py

@@ -45,8 +45,8 @@
 
 Note that ``log_to_file: true`` only means the log *directory* was writable at
 startup. To confirm records are really landing, tail the file over SSH at
-``/cse/web/research/makelab/www[-test]/debug.log`` (the ``/logs/`` URL described in
-docs/DEPLOYMENT.md 404s on both servers).
+``/cse/web/research/makelab/www[-test]/debug.log`` -- there is no web path to the
+log (the old ``/logs/`` URL is gone; see docs/DEPLOYMENT.md).
 """
 
 import json