Add pipe-delimited usecase filtering for Cloud Run deployments

README.md

@@ -61,6 +61,7 @@ export LOGSTORY_CUSTOMER_ID=01234567-0123-4321-abcd-01234567890a
 export LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json
 export LOGSTORY_REGION=US
 export LOGSTORY_AUTO_GET=true  # Auto-download missing usecases
+export LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB  # Filter usecases (pipe-separated)
 
 logstory replay usecase RULES_SEARCH_WORKSHOP
 ```
@@ -74,6 +75,7 @@ LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json
 LOGSTORY_REGION=US
 LOGSTORY_USECASES_BUCKETS=gs://logstory-usecases-20241216,gs://my-custom-bucket
 LOGSTORY_AUTO_GET=true  # Auto-download missing usecases (optional)
+LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB  # Filter usecases (pipe-separated, optional)
 ```
 
 Then run commands without additional options:
@@ -311,6 +313,11 @@ logstory replay all \
   --customer-id=01234567-0123-4321-abcd-01234567890a \
   --credentials-path=/path/to/credentials.json
 
+# Replay specific usecases using environment variable filtering
+LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB logstory replay all \
+  --customer-id=01234567-0123-4321-abcd-01234567890a \
+  --credentials-path=/path/to/credentials.json
+
 # Replay a specific usecase
 logstory replay usecase RULES_SEARCH_WORKSHOP \
   --customer-id=01234567-0123-4321-abcd-01234567890a \
@@ -466,6 +473,7 @@ tree /tmp/var/log/logstory/
 - `LOGSTORY_CREDENTIALS_PATH`: Path to JSON credentials file
 - `LOGSTORY_REGION`: SecOps tenant region (default: US)
 - `LOGSTORY_LOCAL_LOG_DIR`: Base directory for local file output (default: /tmp/var/log/logstory)
+- `LOGSTORY_USECASES`: Filter usecases for `replay all` command (pipe-separated, e.g., `NETWORK_ANALYSIS|GITHUB`)
 
 ### Command Migration Guide
 

docs/configuration.md

@@ -26,6 +26,7 @@ export LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json
 export LOGSTORY_REGION=US
 export LOGSTORY_USECASES_BUCKETS=gs://my-bucket,file:///local/usecases
 export LOGSTORY_AUTO_GET=true  # Auto-download missing usecases
+export LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB  # Filter usecases for 'replay all' command
 
 # Now run commands without additional options
 logstory replay usecase RULES_SEARCH_WORKSHOP  # Will auto-download if missing

docs/env-file.md

@@ -490,6 +490,9 @@ LOGSTORY_REGION=US
 # Supports: gs://bucket, file:///path, bare-bucket-name
 LOGSTORY_USECASES_BUCKETS=gs://bucket1,file:///local/usecases
 
+# Optional: Filter usecases for 'replay all' command (pipe-separated)
+LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB
+
 # Optional: Local file output directory
 LOGSTORY_LOCAL_LOG_DIR=/tmp/var/log/logstory
 

docs/index.md

@@ -460,6 +460,7 @@ Quick start using Makefile targets:
 export LOGSTORY_PROJECT_ID=your-gcp-project-id
 export LOGSTORY_CUSTOMER_ID=your-chronicle-customer-uuid
 export LOGSTORY_API_TYPE=rest  # or 'legacy'
+export LOGSTORY_USECASES=NETWORK_ANALYSIS|GITHUB  # Optional: filter usecases (pipe-separated)
 
 # Deploy to Cloud Run
 make enable-apis

docs/testing-pipe-delimited-usecases.md

@@ -0,0 +1,268 @@
+# Testing Pipe-Delimited Usecase Filtering in Cloud Run
+
+## Overview
+
+This document provides testing instructions for the pipe-delimited usecase filtering functionality in Logstory's Cloud Run deployment. This feature allows you to filter which usecases are processed when using `replay all` by setting the `LOGSTORY_USECASES` environment variable.
+
+## Feature Description
+
+The `LOGSTORY_USECASES` environment variable filtering feature allows you to:
+
+- Use `replay all` command with selective usecase processing
+- Avoid gcloud argument parsing issues entirely
+- Filter usecases using pipe-separated values (e.g., `NETWORK_ANALYSIS|GITHUB`)
+- Maintain backward compatibility - no filtering variable means process all usecases
+
+## Implementation
+
+When `LOGSTORY_USECASES` is set, the `replay all` command:
+1. Parses the pipe-separated usecase list
+2. Validates that all requested usecases exist
+3. Only processes the specified usecases instead of all available ones
+4. Provides clear feedback about which usecases are being processed
+
+## Local Testing
+
+### Prerequisites
+
+Set up your development environment:
+
+```bash
+# Create and activate virtual environment
+python -m venv venv
+source venv/bin/activate
+
+# Install in editable mode
+pip install -e .
+
+# Set required environment variables
+export LOGSTORY_CUSTOMER_ID=your-uuid
+export LOGSTORY_CREDENTIALS_PATH=/path/to/credentials.json
+export LOGSTORY_REGION=US
+export LOGSTORY_API_TYPE=rest
+```
+
+### Test Cases
+
+#### 1. Test Normal Behavior (No Filtering)
+
+```bash
+# Should process all available usecases
+logstory replay all --local-file-output
+
+# Expected output: "Processing all usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP, ..."
+```
+
+#### 2. Test Single Usecase Filtering
+
+```bash
+# Filter to single usecase
+LOGSTORY_USECASES=NETWORK_ANALYSIS logstory replay all --local-file-output
+
+# Expected output: "Processing filtered usecases: NETWORK_ANALYSIS"
+```
+
+#### 3. Test Multiple Usecase Filtering
+
+```bash
+# Filter to multiple usecases
+LOGSTORY_USECASES=NETWORK_ANALYSIS\|RULES_SEARCH_WORKSHOP logstory replay all --local-file-output
+
+# Expected output: "Processing filtered usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP"
+```
+
+#### 4. Test Error Handling
+
+```bash
+# Test with invalid usecase
+LOGSTORY_USECASES=INVALID_USECASE logstory replay all --local-file-output
+
+# Expected: Error message listing available usecases
+```
+
+## Cloud Run Testing
+
+### Prerequisites
+
+Ensure your Cloud Run environment is set up:
+
+```bash
+# Set required environment variables
+export LOGSTORY_API_TYPE=rest
+export LOGSTORY_CUSTOMER_ID=your-uuid
+export LOGSTORY_PROJECT_ID=your-project-id
+export LOGSTORY_REGION=US
+
+# Build and deploy updated Docker image
+make build
+make docker-build
+```
+
+### Test Cases
+
+#### 1. Test Normal Cloud Run Behavior
+
+```bash
+# Process all usecases (no filtering)
+gcloud run jobs execute logstory-replay \
+  --region us-central1 \
+  --args "logstory,replay,all" \
+  --wait
+```
+
+#### 2. Test Single Usecase Filtering
+
+```bash
+# Filter to single usecase
+gcloud run jobs execute logstory-replay \
+  --region us-central1 \
+  --args "logstory,replay,all" \
+  --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS" \
+  --wait
+```
+
+#### 3. Test Multiple Usecase Filtering
+
+```bash
+# Filter to multiple usecases using pipe separator
+gcloud run jobs execute logstory-replay \
+  --region us-central1 \
+  --args "logstory,replay,all" \
+  --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS|RULES_SEARCH_WORKSHOP" \
+  --wait
+```
+
+#### 4. Test with Additional Options
+
+```bash
+# Combine filtering with entities and custom timestamp delta
+gcloud run jobs execute logstory-replay \
+  --region us-central1 \
+  --args "logstory,replay,all,--entities" \
+  --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS,LOGSTORY_TIMESTAMP_DELTA=3d" \
+  --wait
+```
+
+## Validation Methods
+
+### Check Execution Status
+
+```bash
+# Get the latest execution name
+EXECUTION_NAME=$(gcloud run jobs executions list \
+  --job logstory-replay \
+  --region us-central1 \
+  --limit 1 \
+  --format "value(name)")
+
+# Check if it completed successfully
+gcloud run jobs executions describe $EXECUTION_NAME \
+  --region us-central1 \
+  --format "value(status.conditions[0].status)"
+```
+
+### Analyze Logs
+
+```bash
+# Install beta components if needed
+gcloud components install beta --quiet
+
+# View execution logs
+gcloud beta run jobs executions logs read $EXECUTION_NAME --region us-central1
+```
+
+### Success Indicators
+
+Look for these patterns in the logs:
+
+**Filtered Processing:**
+```
+Processing filtered usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP
+Processing usecase: NETWORK_ANALYSIS, logtype: BRO_JSON
+Processing usecase: RULES_SEARCH_WORKSHOP, logtype: POWERSHELL
+Successfully posted entries using RestIngestionBackend
+```
+
+**Normal Processing:**
+```
+Processing all usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP, THW2
+```
+
+**Error Handling:**
+```
+Error: Invalid usecases: INVALID_NAME
+Available usecases: NETWORK_ANALYSIS, RULES_SEARCH_WORKSHOP, THW2
+```
+
+## Troubleshooting
+
+### Common Issues
+
+#### Issue: Container Exits with Error
+**Solution:** Check execution logs for specific error messages and verify environment variables are set correctly.
+
+#### Issue: All Usecases Processed Despite Filtering
+**Solution:** Verify the Docker image was rebuilt and deployed after code changes. Check that `LOGSTORY_USECASES` environment variable is correctly set.
+
+#### Issue: Invalid Usecase Error
+**Solution:** Run `logstory usecases list-installed` to see available usecases, or check the error message for the list of valid options.
+
+### Debugging Commands
+
+```bash
+# Check job configuration
+gcloud run jobs describe logstory-replay --region us-central1
+
+# List recent executions
+gcloud run jobs executions list \
+  --job logstory-replay \
+  --region us-central1 \
+  --limit 5
+
+# Check environment variables in job
+gcloud run jobs describe logstory-replay \
+  --region us-central1 \
+  --format "value(spec.template.template.spec.template.spec.containers[0].env[])"
+```
+
+## Benefits
+
+1. **No gcloud parsing issues** - Environment variables avoid command-line argument parsing problems
+2. **Clean command structure** - Simple `logstory,replay,all` arguments
+3. **Flexible filtering** - Easy to specify any combination of usecases
+4. **Backward compatible** - Existing behavior preserved when no filtering is specified
+5. **Clear feedback** - Logs clearly show which usecases are being processed
+
+## Usage Examples
+
+### Development Workflow
+
+```bash
+# Test locally first
+LOGSTORY_USECASES=NETWORK_ANALYSIS|RULES_SEARCH_WORKSHOP logstory replay all --local-file-output
+
+# Deploy to Cloud Run
+make build && make docker-build
+
+# Test in Cloud Run
+gcloud run jobs execute logstory-replay \
+  --region us-central1 \
+  --args "logstory,replay,all" \
+  --update-env-vars "LOGSTORY_USECASES=NETWORK_ANALYSIS|RULES_SEARCH_WORKSHOP" \
+  --wait
+```
+
+### Production Usage
+
+```bash
+# Set environment variables in .env file or Cloud Run job configuration
+LOGSTORY_USECASES=PRODUCTION_USECASE_1|PRODUCTION_USECASE_2
+
+# Deploy with scheduled execution
+gcloud run jobs execute logstory-replay \
+  --region us-central1 \
+  --args "logstory,replay,all" \
+  --wait
+```
+
+This implementation provides a clean, reliable solution for filtering usecases in Cloud Run deployments while maintaining full backward compatibility and avoiding all gcloud argument parsing issues.

src/logstory/logstory.py

@@ -41,7 +41,7 @@
     except Exception:
       __version__ = "unknown"
     typer.echo(f"logstory {__version__}")
-    raise typer.Exit()
+    raise typer.Exit
 
 
 # Create Typer app and command groups
@@ -112,8 +112,8 @@
   credentials_json = os.getenv("LOGSTORY_CREDENTIALS")
   if credentials_json:
     # Write to temp file and return path
     import json
     import tempfile
 
     try:
       # Validate it's valid JSON
@@ -846,7 +846,7 @@
     ),
     usecases_bucket: str | None = UsecasesBucketOption,
 ):
-  """Replay all usecases."""
+  """Replay all usecases (or filtered usecases if LOGSTORY_USECASES is set)."""
   # Load environment file first (needed for download logic)
   load_env_file(env_file)
 
@@ -892,7 +892,26 @@
         impersonate_service_account,
     )
 
-  usecases = get_usecases()
+  # Get all available usecases
+  all_usecases = get_usecases()
+
+  # Filter usecases if LOGSTORY_USECASES environment variable is set
+  filter_usecases = os.getenv("LOGSTORY_USECASES")
+  if filter_usecases:
+    # Parse pipe-separated list and filter
+    requested_usecases = [uc.strip() for uc in filter_usecases.split("|") if uc.strip()]
+    # Validate that requested usecases exist
+    invalid_usecases = [uc for uc in requested_usecases if uc not in all_usecases]
+    if invalid_usecases:
+      typer.echo(f"Error: Invalid usecases: {', '.join(invalid_usecases)}")
+      typer.echo(f"Available usecases: {', '.join(sorted(all_usecases))}")
+      raise typer.Exit(1)
+    usecases = requested_usecases
+    typer.echo(f"Processing filtered usecases: {', '.join(usecases)}")
+  else:
+    usecases = all_usecases
+    typer.echo(f"Processing all usecases: {', '.join(usecases)}")
+
   _replay_usecases(usecases, "*", entities, timestamp_delta, local_file_output)