GOATS-1013 GOATS-1054 GOATS-1025 GOATS-1053 GOATS-1056: GOA fixes and logging. (#486)

- GOA bug fix for relative path issues when downloading files.
- Add canonical to URL when querying GOA to appropriate URLs.
- Add logging to track download progress and issues.
- Refactor download_goa_files task to use observation record primary key.
- Improve error handling when observation record is not found.
- Add specialized logging to GOATS settings.py template.

Author: davner

docs/changes/486.bugfix.rst

@@ -0,0 +1 @@
+Fixed bug when downloaded folders of data from GOA. GOA now organizes data into structured folders, and the code now correctly handles this structure when downloading and processing the data.

docs/changes/486.change.rst

@@ -0,0 +1 @@
+Added ``/canonical/`` to GOA URL paths as specified in GOA API documentation.

docs/changes/486.new.rst

@@ -0,0 +1 @@
+Added rotating log file to GOATS so users can keep track of operations over time and better report issues. Log files are stored in the main GOATS directory as "goats.log", with a maximum size of 50 MB and up to two backup files. The console logging remains for real-time feedback during execution.

src/goats_setup/templates/goats_setup/settings.tmpl

@@ -208,12 +208,37 @@ MEDIA_URL = "/data/"
 LOGGING = {
     "version": 1,
     "disable_existing_loggers": False,
+    "formatters": {
+        "verbose": {
+            "format": "{levelname} {asctime} {name} {filename}:{lineno} - {message}",
+            "style": "{",
+        },
+        "simple": {
+            "format": "{levelname} {message}",
+            "style": "{",
+        },
+    },
     "handlers": {
         "console": {
             "class": "logging.StreamHandler",
-        }
+            "formatter": "simple",
+        },
+        "file": {
+            "class": "logging.handlers.RotatingFileHandler",
+            "filename": BASE_DIR / "goats.log",
+            "formatter": "verbose",
+            "maxBytes": 1024 * 1024 * 50,  # 50 MB.
+            "backupCount": 2,  # Keep two backup log files.
+        },
+    },
+    "loggers": {
+        "": {"handlers": ["console", "file"], "level": "INFO"},
+        "goats_tom": {
+            "handlers": ["console", "file"],
+            "level": "DEBUG",
+            "propagate": False,
+        },
     },
-    "loggers": {"": {"handlers": ["console"], "level": "INFO"}},
 }
 
 # Caching