wasm demo script + filtering proxy

Author: wdbaruni

edge/wasm-sensors/Makefile

@@ -62,29 +62,38 @@ push-puller: ## Push the puller Docker image for all architectures
 		--push \
 		-f containers/docker/sqs-puller/Dockerfile containers/docker/sqs-puller
 
-
 ##@ WASM
 
-build-wasm: build-sqs-publisher build-config-updater build-config-reader build-dir-lister
+build-wasm: build-sqs-publisher
+
 build-sqs-publisher: ## Build the SQS publisher WASM module
 	@echo "Building SQS publisher WASM module..."
 	@cd containers/wasm/sqs-publisher && tinygo build -o ../../../network/modules/publisher.wasm -target wasi .
 	@echo "SQS publisher WASM build completed successfully!"
 
-build-config-updater: ## Build the config updater WASM module
-	@echo "Building config updater WASM module..."
-	@cd containers/wasm/config-updater && tinygo build -o ../../../network/modules/config-updater.wasm -target wasi .
-	@echo "Config updater WASM build completed successfully!"
+##@ Legacy WASM
+
+build-legacy: build-legacy-config-updater build-legacy-config-reader build-legacy-dir-lister build-legacy-sqs-publisher
+
+build-legacy-config-updater: ## Build the legacy config updater WASM module
+	@echo "Building legacy config updater WASM module..."
+	@cd containers/wasm/legacy/config-updater && tinygo build -o ../../../../network/modules/legacy/config-updater.wasm -target wasi .
+	@echo "Legacy config updater WASM build completed successfully!"
+
+build-legacy-config-reader: ## Build the legacy config reader WASM module
+	@echo "Building legacy config reader WASM module..."
+	@cd containers/wasm/legacy/config-reader && tinygo build -o ../../../../network/modules/legacy/config-reader.wasm -target wasi .
+	@echo "Legacy config reader WASM build completed successfully!"
 
-build-config-reader: ## Build the config reader WASM module
-	@echo "Building config reader WASM module..."
-	@cd containers/wasm/config-reader && tinygo build -o ../../../network/modules/config-reader.wasm -target wasi .
-	@echo "Config reader WASM build completed successfully!"
+build-legacy-dir-lister: ## Build the legacy directory lister WASM module
+	@echo "Building legacy directory lister WASM module..."
+	@cd containers/wasm/legacy/dir-lister && tinygo build -o ../../../../network/modules/legacy/dir-lister.wasm -target wasi .
+	@echo "Legacy directory lister WASM build completed successfully!"
 
-build-dir-lister: ## Build the directory lister WASM module
-	@echo "Building directory lister WASM module..."
-	@cd containers/wasm/dir-lister && tinygo build -o ../../../network/modules/dir-lister.wasm -target wasi .
-	@echo "Directory lister WASM build completed successfully!"
+build-legacy-sqs-publisher: ## Build the legacy SQS publisher WASM module
+	@echo "Building legacy SQS publisher WASM module..."
+	@cd containers/wasm/legacy/sqs-publisher && tinygo build -o ../../../../network/modules/legacy/sqs-publisher.wasm -target wasi .
+	@echo "Legacy SQS publisher WASM build completed successfully!"
 
 ##@ Development
 

edge/wasm-sensors/README.md

@@ -16,38 +16,46 @@ The WASM Sensors project demonstrates:
 The Docker Compose network setup establishes a comprehensive Bacalhau environment with the following components:
 
 ### Orchestrator
+
 - Central coordinator for the Bacalhau network
 - Manages job scheduling and distribution
 - Exposes NATS port (4222) for communication with compute nodes
 - Exposes WebUI port (8438) for monitoring and management
 
 ### Edge Compute Nodes
+
 The setup includes multiple edge compute nodes organized by geographical regions:
+
 - **edge-us**: Edge nodes simulating US region deployment
 - **edge-eu**: Edge nodes simulating European region deployment
 - **edge-as**: Edge nodes simulating Asian region deployment
 
 Each edge compute node:
+
 - Is labeled with its region and type (e.g., `region=us,type=edge`)
 - Has access to WASM modules from the host system
 - Connects to the orchestrator for job coordination
 
 ### Web Services Compute Node
+
 - Dedicated node for running centralized web services
 - Labeled as `type=web-services`
 - Runs in privileged mode to support Docker-in-Docker
 - Hosts the SQS Proxy and SQS Puller when deployed as Bacalhau service jobs
 
 ### Supporting Services
+
 - **SQS Proxy**: Direct service running on port 9090
 - **SQS Puller**: Direct service processing messages from SQS
 
 ### Client Node
+
 - Interactive shell for interacting with the Bacalhau network
 - Configured to connect to the orchestrator
 - Useful for submitting jobs and monitoring status
 
 ### Network Diagram
+
 ```
 ┌─────────────────────────────────────────────────────────────────┐
 │                      bacalhau-edge Network                      │
@@ -64,7 +72,7 @@ Each edge compute node:
 │  ┌─────┴─────┐                               ┌──────┴───────┐   │
 │  │           │                               │              │   │
 │  │  Client   │                               │  SQS Proxy   │   │
-│  │           │                               │   (9090)     │   │
+│  │           │                               │  (8080/9090) │   │
 │  └───────────┘                               └──────────────┘   │
 │                                                     ▲           │
 │                                                     │           │
@@ -85,66 +93,78 @@ Each edge compute node:
 The SQS Publisher is a WebAssembly module that generates simulated sensor data and sends it to the SQS Proxy. It can be configured with parameters:
 
 - **Interval**: Time between messages (in seconds)
-- **Color**: Hex color code for visual identification
-- **RandomOff**: Whether to use deterministic or random icons
+- **Color**: Hex color code for visual identification or "-1" for random color
+- **Emoji**: Index of emoji to use (-1 for random)
 
-It reads its configuration from a YAML file that can be updated while the job is running.
+The configuration is set at startup and remains constant throughout the job's execution.
 
 ### Message Schema
 
 The SQS Publisher sends messages with the following JSON schema:
 
 ```json
 {
-  "job_id": "bacalhau-job-id",        // The Bacalhau job ID of the publisher
-  "execution_id": "execution-id",      // The Bacalhau execution ID 
-  "icon_name": "🚀",                   // An emoji representing the message
+  "job_id": "bacalhau-job-id", // The Bacalhau job ID of the publisher
+  "execution_id": "execution-id", // The Bacalhau execution ID
+  "hostname": "edge-node-1", // The hostname of the edge node
+  "job_submission_time": 1711449600, // Unix timestamp of when the Bacalhau job was submitted
+  "icon_name": "😀", // An emoji from the supported list
   "timestamp": "2025-03-26T12:34:56Z", // ISO 8601 timestamp in UTC
-  "color": "#FF5733",                  // Hex color code from configuration
-  "sequence": 42                       // Sequential message counter
+  "color": "#FF5733", // Hex color code
+  "sequence": 42, // Sequential message counter
+  "region": "us" //  Region from environment
 }
 ```
 
 ### Message Fields:
 
 - **job_id**: The Bacalhau job ID that identifies the publisher job
 - **execution_id**: The Bacalhau execution ID for the specific execution instance
-- **icon_name**: An emoji selected from a predefined list (random or fixed depending on the `randomOff` configuration)
+- **hostname**: The hostname of the edge node sending the message
+- **job_submission_time**: Unix timestamp of when the Bacalhau job was submitted
+- **icon_name**: An emoji from a predefined list of supported emojis
 - **timestamp**: ISO 8601 formatted timestamp in UTC
-- **color**: Hex color code specified in the configuration
+- **color**: Hex color code (can be specified or random)
 - **sequence**: Incrementing counter for each message sent by this publisher instance
+- **region**: AWS region read from the REGION environment variable
+
+### Legacy Implementation
+
+The `legacy` directory contains an initial implementation attempt to support updating configuration files at runtime using WebAssembly. This approach was ultimately not pursued due to limitations with the current wazero-based executor. Instead, we now use direct job configuration updates.
 
-These messages are sent to the SQS Proxy, which forwards them to Amazon SQS. The SQS Puller then retrieves and processes these messages from the queue.
+The legacy implementation included:
+
+- `config-reader`: Attempted to read configuration files at runtime
+- `config-updater`: Attempted to update configuration files at runtime
+- `dir-lister`: Utility for listing directory contents
+- `sqs-publisher`: Previous version of the message publisher
 
 ### SQS Proxy
 
 The SQS Proxy is a FastAPI application that:
+
 - Receives HTTP POST requests from edge nodes
 - Forwards valid JSON messages to Amazon SQS
 - Provides a health check endpoint
 - Can be deployed either as a Bacalhau service job or alongside the Docker Compose setup
 
-**Important Note**: The SQS Proxy is available at two different ports:
-- Port `8080` when deployed as a Bacalhau service job
-- Port `9090` when deployed via Docker Compose
+**Important Note**: The SQS Proxy URL varies depending on the deployment method:
+
+- When deployed via Docker Compose: `http://bacalhau-edge-sqs-proxy-1:9090`
+- When deployed as a Bacalhau service job: `http://bacalhau-edge-web-services-1:8080`
+
+Make sure to use the appropriate URL when deploying the SQS Publisher jobs.
 
 ### SQS Puller
 
 The SQS Puller retrieves messages from Amazon SQS and processes them. In this demo, it simply logs messages to stdout but could be extended to:
+
 - Process data
 - Trigger alerts
 - Store data in a database
 
 Like the SQS Proxy, it can be deployed either as a Bacalhau service job or alongside Docker Compose.
 
-### Config Updater
-
-The Config Updater is a WASM module that can update the SQS Publisher's configuration file. Due to limitations in Bacalhau v1.7.0, it cannot directly update a running job's configuration. To apply changes:
-
-1. Use Config Updater to modify the configuration file
-2. Stop the current SQS Publisher job
-3. Deploy a new SQS Publisher job that will use the updated configuration
-
 ## 🚀 Getting Started
 
 ### Prerequisites
@@ -158,6 +178,7 @@ The Config Updater is a WASM module that can update the SQS Publisher's configur
 ### Initial Setup
 
 1. Clone the repository:
+
    ```bash
    git clone https://github.com/bacalhau-project/wasm-sensors.git
    cd wasm-sensors
@@ -166,58 +187,131 @@ The Config Updater is a WASM module that can update the SQS Publisher's configur
 2. Create an SQS queue in your AWS account
 
 3. Configure your environment:
+
    ```bash
    cp network/.env.template network/.env
    ```
+
    Edit `network/.env` and add your AWS credentials and SQS queue URL.
 
-4. Start the Bacalhau network:
+### Complete Demo Scenario
+
+This scenario demonstrates the message flow and deployment behavior across different regions:
+
+1. Start the network:
+
+   ```bash
+   ./demo.sh start_network
+   ```
+
+2. Deploy to US region with red color and rocket emoji:
+
+   ```bash
+   ./demo.sh deploy "us" "#FF0000" 0
+   ```
+
+   This will deploy the SQS Publisher to US edge nodes with red color and rocket emoji 🚀.
+
+3. Deploy to EU region with blue color and satellite emoji:
+
+   ```bash
+   ./demo.sh deploy "eu" "#0000FF" 1
+   ```
+
+   This will deploy the SQS Publisher to EU edge nodes with blue color and satellite emoji 📡.
+
+4. Deploy to AS region with green color and light bulb emoji:
+
+   ```bash
+   ./demo.sh deploy "as" "#00FF00" 2
+   ```
+
+   This will deploy the SQS Publisher to AS edge nodes with green color and light bulb emoji 💡.
+
+5. Update US deployment with new configuration (purple color and lightning emoji):
+
    ```bash
-   cd network
-   docker-compose up -d
+   ./demo.sh deploy "us" "#800080" 3
    ```
 
+   This will deploy a new version of the SQS Publisher to US nodes with purple color and lightning emoji ⚡.
+
+6. Disconnect EU region:
+
+   ```bash
+   ./demo.sh disconnect "eu"
+   ```
+
+   This will stop the EU deployment.
+
+7. Deploy new configuration to EU (yellow color and battery emoji):
+
+   ```bash
+   ./demo.sh deploy "eu" "#FFFF00" 4
+   ```
+
+   This will deploy a new version of the SQS Publisher to EU nodes with yellow color and battery emoji 🔋.
+
+8. Reconnect EU region:
+
+   ```bash
+   ./demo.sh reconnect "eu"
+   ```
+
+   This will restart the EU deployment.
+
+9. Monitor message flow:
+   ```bash
+   ./demo.sh logs
+   ```
+   You should see:
+   - Messages from US nodes (new deployment with purple color and lightning emoji ⚡)
+   - Messages from EU nodes (new deployment with yellow color and battery emoji 🔋)
+   - Messages from AS nodes (green color and light bulb emoji 💡)
+
+This demo shows how to deploy and update the SQS Publisher across different regions, with each region having its own unique visual identity through colors and emojis.
+
 ### Deployment Options
 
 #### Option 1: Deploy using Docker Compose Services
 
 The provided Docker Compose file already includes the SQS Proxy and SQS Puller services, running on ports:
+
 - SQS Proxy: `9090`
 - SQS Puller: logs to Docker Compose logs
 
 #### Option 2: Deploy using Bacalhau Service Jobs
 
 Deploy the SQS Proxy as a Bacalhau service job:
+
 ```bash
 bacalhau job run -V queue_url=wasm-sensors-demo jobs/sqs-proxy.yaml
 ```
 
 Deploy the SQS Puller as a Bacalhau service job:
+
 ```bash
 bacalhau job run -V queue_url=wasm-sensors-demo jobs/sqs-puller.yaml
 ```
 
 ### Deploy the SQS Publisher to Edge Nodes
 
 Deploy the SQS Publisher WebAssembly module targeting the Docker Compose SQS Proxy:
-```bash
-bacalhau job run -V proxy=http://bacalhau-edge-sqs-proxy-1:9090 jobs/sqs-publisher.yaml
-```
 
-To target the Bacalhau service job SQS Proxy instead:
 ```bash
-bacalhau job run -V proxy=http://bacalhau-edge-web-services-1:8080 jobs/sqs-publisher.yaml
+# With specific color and emoji
+bacalhau job run -V proxy=http://bacalhau-edge-sqs-proxy-1:9090 -V color="#FF5733" -V emoji=0 jobs/sqs-publisher.yaml
+
+# With random color and emoji
+bacalhau job run -V proxy=http://bacalhau-edge-sqs-proxy-1:9090 -V color="-1" -V emoji=-1 jobs/sqs-publisher.yaml
 ```
 
-### Update Configuration
+To target the Bacalhau service job SQS Proxy instead:
 
-To update the configuration (note: currently doesn't work with Bacalhau v1.7.0):
 ```bash
-bacalhau job run -V color="#FEFEFE" jobs/config-updater.yaml
+bacalhau job run -V proxy=http://bacalhau-edge-web-services-1:8080 -V color="#FF5733" -V emoji=0 jobs/sqs-publisher.yaml
 ```
 
-As noted earlier, you'll need to restart the SQS Publisher job for these changes to take effect.
-
 ## 🛠️ Development
 
 ### Development Notes
@@ -275,8 +369,9 @@ curl http://localhost:9090/health
 ```
 
 This should return:
+
 ```json
-{"status":"healthy"}
+{ "status": "healthy" }
 ```
 
 ## 🔄 Message Flow
@@ -289,6 +384,5 @@ This should return:
 
 ## ⚠️ Known Limitations
 
-- **Configuration Updates**: Due to Bacalhau v1.7.0 limitations, you cannot dynamically update a running job's configuration. You need to stop and redeploy the job.
+- **Configuration Updates**: Configuration is set at job startup and cannot be changed during execution. To change configuration, you need to stop and redeploy the job.
 - **Direct File Access**: WASM modules have limited filesystem access through Bacalhau.
-