Add daily smoke tests for staging and prod (#442)

* feat: add daily smoke tests workflow and e2e testing script

* fix: harden smoke tests with timeouts, error handling, and workflow fixes

Address PR review feedback:
- Add request timeouts (30s default, 90s streaming) to prevent indefinite hangs
- Add streaming chunk limits (1000 max) as safety cap
- Catch request exceptions in HTTP helpers so individual failures don't crash the suite
- Wrap each test in run_test() to isolate unhandled exceptions
- Mask API key in output to prevent partial secret leak in CI logs
- Validate API_KEY is set before running tests
- Fix workflow matrix exclude (broken with include-only matrix) using job-level if
- Add job timeout-minutes: 15 to prevent 6-hour workflow hangs
- Add API key validation step in workflow
- Update actions/checkout@v4 to v6 for consistency
- Fix Datadog curl to show errors instead of failing silently
- Use tempfile instead of writing test files to CWD
- Add missing expect_status to report() calls
- Track conversation creation and cleanup deletes via report()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: Evrard-Nil

.github/workflows/daily-smoke-test.yml

@@ -0,0 +1,82 @@
+name: Daily Smoke Tests
+
+on:
+  schedule:
+    # Every day at 06:00 UTC
+    - cron: "0 6 * * *"
+  workflow_dispatch:
+    inputs:
+      environment:
+        description: "Environment to test"
+        required: true
+        default: both
+        type: choice
+        options:
+          - both
+          - staging
+          - prod
+
+jobs:
+  smoke-test:
+    name: Smoke test (${{ matrix.env_name }})
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - env_name: staging
+            api_base_url: https://cloud-stg-api.near.ai/v1
+            api_key_secret: SMOKE_TEST_API_KEY_STAGING
+          - env_name: prod
+            api_base_url: https://cloud-api.near.ai/v1
+            api_key_secret: SMOKE_TEST_API_KEY_PROD
+    if: >-
+      github.event_name == 'schedule'
+      || inputs.environment == 'both'
+      || inputs.environment == matrix.env_name
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: pip install requests
+
+      - name: Validate API key exists
+        env:
+          API_KEY: ${{ secrets[matrix.api_key_secret] }}
+        run: |
+          if [ -z "$API_KEY" ]; then
+            echo "ERROR: Secret ${{ matrix.api_key_secret }} is not configured"
+            exit 1
+          fi
+
+      - name: Run smoke tests against ${{ matrix.env_name }}
+        env:
+          API_BASE_URL: ${{ matrix.api_base_url }}
+          API_KEY: ${{ secrets[matrix.api_key_secret] }}
+        run: python tests/e2e.py
+
+      - name: Page via Datadog on failure
+        if: failure()
+        run: |
+          curl -f -X POST "https://api.datadoghq.com/api/v1/events" \
+            -H "DD-API-KEY: ${{ secrets.DD_API_KEY }}" \
+            -H "Content-Type: application/json" \
+            -d '{
+              "title": "Cloud API smoke tests failed (${{ matrix.env_name }})",
+              "text": "Daily smoke tests failed for ${{ matrix.env_name }}.\n\nWorkflow run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}",
+              "priority": "normal",
+              "alert_type": "error",
+              "tags": [
+                "service:cloud-api",
+                "env:${{ matrix.env_name }}",
+                "source:github-actions",
+                "test:smoke"
+              ]
+            }' || echo "WARNING: Failed to post Datadog event"