feat: plugin fetch CLI, dotenv support, location assumptions plugin, and plugin READMEs (#47)

- Add `python autobiographer.py fetch <plugin>` CLI subcommand; fetchable
  plugins download data, non-fetchable ones print manual export steps
- Add python-dotenv support so .env is loaded automatically by both the
  CLI and Streamlit app — no more manual export in every terminal session
- Add Fetch Latest Data button in sidebar with live per-page progress
  shown in a sidebar-level placeholder (always visible, even when the
  plugin expander is collapsed)
- Show fetch identity (username) and save path before fetching
- Auto-populate config field after a successful fetch
- Auto-expand plugin expander and show warning when saved path no longer
  exists on disk
- Extract location assumptions into its own AssumptionsPlugin
  (location-context type) with its own sidebar section and config field;
  remove assumptions_file from SwarmPlugin
- Add pytest pythonpath = ["."] so tests run without AUTOBIO_PYTHONPATH
- Add per-plugin READMEs (lastfm, swarm, assumptions) and a Data Sources
  table in the main README linking to all three

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: jschloman

.env.example

@@ -1,5 +1,15 @@
-# Copy to .env and fill in your values, then:
-#   docker compose run --rm dashboard python autobiographer.py --user YOUR_USERNAME
+# Copy this file to .env and fill in your values.
+#
+# The app and CLI load .env automatically — no manual `export` needed.
+#
+# Fetch your Last.fm listening history:
+#   python autobiographer.py fetch lastfm
+#
+# Or from Docker:
+#   docker compose run --rm dashboard python autobiographer.py fetch lastfm
+#
+# Get a Last.fm API key at: https://www.last.fm/api/account/create
+
 AUTOBIO_LASTFM_API_KEY=your_api_key_here
 AUTOBIO_LASTFM_API_SECRET=your_api_secret_here
 AUTOBIO_LASTFM_USERNAME=your_username_here

README.md

@@ -24,6 +24,18 @@ Autobiographer is a Python-based toolkit that allows you to fetch, store, and ex
 
 <img src="assets/flythrough.gif" style="width: 500">
 
+## Data Sources
+
+Autobiographer is built around a plugin system. Each plugin owns a specific data source — its format, how to obtain it, and how it maps into the common schema. All path configuration happens in the sidebar; nothing is hardcoded.
+
+| Plugin | Type | Description |
+|--------|------|-------------|
+| [Last.fm Music History](plugins/sources/lastfm/README.md) | what-when | Your complete listening history, fetched automatically via the Last.fm API. One click in the sidebar downloads everything. |
+| [Foursquare / Swarm Check-ins](plugins/sources/swarm/README.md) | where-when | Your check-in history from the Swarm app. Requires a one-time manual data export request from Foursquare. |
+| [Location Assumptions](plugins/sources/assumptions/README.md) | location-context | A user-authored JSON file that fills in your location for periods not covered by Swarm — trips, recurring holidays, and home residency rules. |
+
+Each plugin has its own README with setup instructions, data format details, and schema documentation.
+
 ## Quickstart (Docker)
 
 No Python knowledge required — just [Docker](https://www.docker.com/products/docker-desktop/).
@@ -45,7 +57,7 @@ To download your listening history directly into the mounted data volume, pass y
 ```bash
 cp .env.example .env           # fill in your API key, secret, and username
 docker compose run --rm dashboard \
-    python autobiographer.py --user YOUR_USERNAME
+    python autobiographer.py fetch lastfm
 docker compose up
 ```
 
@@ -79,21 +91,45 @@ docker compose up
 
 ### 3. Configuration
 
-Set your Last.fm credentials in your environment:
+Set your credentials as environment variables. Required for Last.fm fetching:
 ```bash
 export AUTOBIO_LASTFM_API_KEY="your_api_key"
 export AUTOBIO_LASTFM_API_SECRET="your_api_secret"
 export AUTOBIO_LASTFM_USERNAME="your_username"
 ```
 
+Get a Last.fm API key at: https://www.last.fm/api/account/create
+
 ### 4. Usage
 
 #### Fetch Your Data
-Download your listening history to a local CSV file:
+
+The `fetch` command targets a specific plugin. Plugins that support automatic download
+will retrieve your data; plugins that require a manual export will print step-by-step
+instructions instead.
+
 ```bash
-python autobiographer.py --user your_username
+# Download Last.fm listening history (requires env vars above)
+python autobiographer.py fetch lastfm
+
+# Print Foursquare/Swarm manual export instructions
+python autobiographer.py fetch swarm
+
+# Limit to recent pages or a date range
+python autobiographer.py fetch lastfm --pages 5
+python autobiographer.py fetch lastfm --from-date 2024-01-01 --to-date 2024-12-31
+
+# Save to a custom location
+python autobiographer.py fetch lastfm --output data/my_tracks.csv
 ```
 
+If a plugin is not yet configured the command will list exactly which environment
+variables are missing and what each one is for.
+
+The app sidebar also exposes this directly: a **Fetch Latest Data** button appears for
+plugins that support automatic retrieval, and step-by-step manual instructions are
+shown for plugins that require a data export from the provider.
+
 #### Launch the Streamlit Dashboard
 Start the interactive Streamlit application:
 ```bash
@@ -205,7 +241,7 @@ Additional utility scripts are available in the `tools/` directory:
 ## Project Structure
 
 ```
-autobiographer.py       # Last.fm API fetch + data save CLI
+autobiographer.py       # Data-fetching CLI (`fetch <plugin>`) + Last.fm API client
 visualize.py            # Streamlit dashboard (assembles views from plugins)
 export_html.py          # Static HTML export — single self-contained report file
 analysis_utils.py       # Shared data processing and caching logic
@@ -215,8 +251,9 @@ plugins/
   sources/
     base.py             # SourcePlugin ABC + validate_schema()
     __init__.py         # REGISTRY + @register decorator + load_builtin_plugins()
-    lastfm/loader.py    # Last.fm source plugin
-    swarm/loader.py     # Foursquare/Swarm source plugin
+    lastfm/loader.py        # Last.fm source plugin
+    swarm/loader.py         # Foursquare/Swarm source plugin
+    assumptions/loader.py   # Location assumptions plugin
 notebooks/              # Jupyter notebooks for custom analysis
 tools/                  # Utility scripts (audio muxing, etc.)
 data/                   # Local data storage (CSVs, cache, Swarm JSON exports)
@@ -389,6 +426,7 @@ Selected paths are persisted to `data/config.json` so they survive application r
 |---|---|
 | `what-when` | `timestamp`, `label`, `sublabel`, `category`, `source_id` |
 | `where-when` | `timestamp`, `lat`, `lng`, `place_name`, `place_type`, `source_id` |
+| `location-context` | No required columns — defines enrichment data, not a primary stream |
 
 `validate_schema()` raises `ValueError` at load time if any required column is absent.
 

autobiographer.py

@@ -1,10 +1,13 @@
 import argparse
 import os
 import time
-from typing import Optional
+from typing import Callable, Optional
 
 import pandas as pd
 import requests
+from dotenv import load_dotenv
+
+load_dotenv()
 
 
 class Autobiographer:
@@ -30,10 +33,26 @@ def fetch_recent_tracks(
         pages: Optional[int] = None,
         from_ts: Optional[int] = None,
         to_ts: Optional[int] = None,
+        progress_callback: Optional[Callable[[int, int], None]] = None,
     ) -> list[dict]:
-        """Fetch recent tracks for the user."""
+        """Fetch recent tracks for the user.
+
+        Args:
+            limit: Tracks per API page (max 200).
+            pages: Stop after this many pages; None fetches all.
+            from_ts: Unix timestamp lower bound (inclusive).
+            to_ts: Unix timestamp upper bound (inclusive).
+            progress_callback: Optional callable invoked after each page with
+                ``(current_page, total_pages)`` so callers can report progress.
+                Also receives ``(0, total_pages)`` before the first page fetch
+                once the total is known.
+
+        Returns:
+            List of raw track dicts from the Last.fm API.
+        """
         all_tracks = []
         current_page = 1
+        total_pages = 1
 
         while True:
             print(f"Fetching page {current_page}...")
@@ -54,6 +73,9 @@ def fetch_recent_tracks(
             all_tracks.extend(tracks)
 
             total_pages = int(data.get("recenttracks", {}).get("@attr", {}).get("totalPages", 1))
+            if progress_callback:
+                progress_callback(current_page, total_pages)
+
             if pages and current_page >= pages:
                 break
             if current_page >= total_pages:
@@ -87,52 +109,152 @@ def save_tracks_to_csv(self, tracks: list[dict], filename: Optional[str] = None)
         print(f"Saved {len(df)} tracks to {filename}")
 
 
-def main() -> None:
-    parser = argparse.ArgumentParser(description="Fetch Last.fm listening history.")
-    parser.add_argument(
-        "--user", help="Last.fm username (defaults to AUTOBIO_LASTFM_USERNAME env var)"
-    )
-    parser.add_argument("--pages", type=int, help="Limit number of pages to fetch")
-    parser.add_argument("--from_date", help="Start date (YYYY-MM-DD)")
-    parser.add_argument("--to_date", help="End date (YYYY-MM-DD)")
+def _parse_date(date_str: str, label: str) -> Optional[int]:
+    """Parse a YYYY-MM-DD date string and return a Unix timestamp.
+
+    Args:
+        date_str: Date string in YYYY-MM-DD format.
+        label: Human-readable label used in error messages (e.g. "from_date").
+
+    Returns:
+        Unix timestamp as int, or None if date_str is empty.
+
+    Raises:
+        SystemExit: If the date string is not in the expected format.
+    """
+    if not date_str:
+        return None
+    try:
+        return int(time.mktime(time.strptime(date_str, "%Y-%m-%d")))
+    except ValueError as exc:
+        print(f"Error: Invalid {label} format '{date_str}'. Use YYYY-MM-DD.")
+        raise SystemExit(1) from exc
 
-    args = parser.parse_args()
 
-    api_key = os.getenv("AUTOBIO_LASTFM_API_KEY")
-    api_secret = os.getenv("AUTOBIO_LASTFM_API_SECRET")
-    username = args.user or os.getenv("AUTOBIO_LASTFM_USERNAME")
+def _run_fetch(args: argparse.Namespace) -> None:
+    """Execute the ``fetch`` subcommand.
 
-    if not all([api_key, api_secret, username]):
+    Routes to the named plugin's fetch() method if it supports programmatic
+    retrieval, or prints manual download instructions if it does not.
+
+    Args:
+        args: Parsed CLI arguments including ``plugin``, ``output``, ``pages``,
+            ``from_date``, and ``to_date``.
+    """
+    from plugins.sources import REGISTRY, load_builtin_plugins
+
+    load_builtin_plugins()
+
+    plugin_id: str = args.plugin
+    if plugin_id not in REGISTRY:
+        available = ", ".join(sorted(REGISTRY))
+        print(f"Error: Unknown plugin '{plugin_id}'. Available plugins: {available}")
+        raise SystemExit(1)
+
+    plugin = REGISTRY[plugin_id]()
+
+    if not plugin.FETCHABLE:
+        print(f"{plugin.DISPLAY_NAME} does not support automatic fetching.\n")
+        print(plugin.get_manual_download_instructions())
+        return
+
+    # Validate env vars before attempting the fetch.
+    env_vars = plugin.get_fetch_env_vars()
+    missing = [v for v in env_vars if not os.getenv(v["var"])]
+    if missing:
+        print("Error: Missing required configuration for fetching. Set the following:\n")
+        for v in missing:
+            print(f"  {v['var']}: {v['description']}")
         print(
-            "Error: AUTOBIO_LASTFM_API_KEY, AUTOBIO_LASTFM_API_SECRET, and "
-            "AUTOBIO_LASTFM_USERNAME must be set."
+            f"\nThen re-run: python autobiographer.py fetch {plugin_id}\n"
+            "See README.md for full configuration instructions."
         )
-        return
+        raise SystemExit(1)
 
-    from_ts = None
-    if args.from_date:
-        try:
-            # Beginning of the day
-            from_ts = int(time.mktime(time.strptime(args.from_date, "%Y-%m-%d")))
-        except ValueError:
-            print(f"Error: Invalid from_date format '{args.from_date}'. Use YYYY-MM-DD.")
-            return
-
-    to_ts = None
-    if args.to_date:
-        try:
-            # End of the day (23:59:59)
-            to_struct = time.strptime(args.to_date, "%Y-%m-%d")
-            to_ts = int(time.mktime(to_struct)) + 86399
-        except ValueError:
-            print(f"Error: Invalid to_date format '{args.to_date}'. Use YYYY-MM-DD.")
-            return
-
-    if not api_key or not api_secret or not username:
-        return
-    visualizer = Autobiographer(api_key, api_secret, username)
-    tracks = visualizer.fetch_recent_tracks(pages=args.pages, from_ts=from_ts, to_ts=to_ts)
-    visualizer.save_tracks_to_csv(tracks)
+    from_ts = _parse_date(args.from_date or "", "from_date")
+    to_ts_raw = _parse_date(args.to_date or "", "to_date")
+    # Shift to-date to end of day so the full day is included.
+    to_ts = to_ts_raw + 86399 if to_ts_raw is not None else None
+
+    print(f"Fetching {plugin.DISPLAY_NAME} data...")
+    plugin.fetch(
+        output_path=args.output or None,
+        pages=args.pages,
+        from_ts=from_ts,
+        to_ts=to_ts,
+    )
+
+
+def main() -> None:
+    """Entry point for the Autobiographer data-fetching CLI.
+
+    Subcommands
+    -----------
+    fetch <plugin>
+        Fetch data for the named plugin. For plugins that support programmatic
+        retrieval (e.g. ``lastfm``) this downloads and saves data locally.
+        For manual-export plugins (e.g. ``swarm``) this prints step-by-step
+        instructions for obtaining the data from the provider.
+
+    Examples
+    --------
+    ::
+
+        python autobiographer.py fetch lastfm
+        python autobiographer.py fetch lastfm --pages 5
+        python autobiographer.py fetch lastfm --from-date 2024-01-01
+        python autobiographer.py fetch swarm
+    """
+    parser = argparse.ArgumentParser(
+        description="Autobiographer data-fetching CLI.",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+    )
+    subparsers = parser.add_subparsers(dest="command", metavar="COMMAND")
+
+    fetch_parser = subparsers.add_parser(
+        "fetch",
+        help="Fetch or get instructions for a plugin's data.",
+        description=(
+            "Fetch data for a registered source plugin. "
+            "Plugins that support automatic retrieval will download and save data. "
+            "Plugins that require a manual export will print step-by-step instructions."
+        ),
+    )
+    fetch_parser.add_argument(
+        "plugin",
+        metavar="PLUGIN",
+        help="Plugin ID to target (e.g. lastfm, swarm).",
+    )
+    fetch_parser.add_argument(
+        "--output",
+        metavar="PATH",
+        help="Output file or directory path (overrides the plugin's default location).",
+    )
+    fetch_parser.add_argument(
+        "--pages",
+        type=int,
+        metavar="N",
+        help="Limit to N pages of results (Last.fm only).",
+    )
+    fetch_parser.add_argument(
+        "--from-date",
+        dest="from_date",
+        metavar="YYYY-MM-DD",
+        help="Only fetch records on or after this date.",
+    )
+    fetch_parser.add_argument(
+        "--to-date",
+        dest="to_date",
+        metavar="YYYY-MM-DD",
+        help="Only fetch records on or before this date.",
+    )
+
+    args = parser.parse_args()
+
+    if args.command == "fetch":
+        _run_fetch(args)
+    else:
+        parser.print_help()
 
 
 if __name__ == "__main__":