Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
145 lines (106 loc) · 5.61 KB

DEVELOPMENT.md

File metadata and controls

145 lines (106 loc) · 5.61 KB
Raw

Development Guide

Deployment

Deploying as Web App

  1. In Apps Script, go to Deploy > New Deployment
  2. Select Web App and set access to "Anyone" or "Anyone with the link"
  3. Save the deployment URL for the /exec endpoint

Trigger Execution

  • Use the /exec endpoint URL for manual or API triggers
    • Example: https://script.google.com/macros/s/{SCRIPT_ID}/exec
  • The /dev route uses the latest saved code (no redeployment needed)

Time-Driven Triggers

In Apps Script, go to Triggers > Add Trigger and configure periodic execution (e.g., every 15 minutes, hourly, or daily).

Testing

Sanity Tests

  • Tag test emails with the Txs/[test tube emoji] label
  • Run testDoGet() or hit the /dev endpoint
  • Compare test results in the TestResults sheet with expected outputs in TestsExpected
  • The compareRows() function highlights mismatches and missing rows

Test with Specific Emails

Use getTestThreadsOrQuery() in 5-test.gs to target specific emails:

function getTestThreadsOrQuery() {
    // Return email thread IDs for targeted testing
    return ['thread_id_1', 'thread_id_2'];

    // Or return a Gmail search query
    // return 'subject:(Transaction SMS [12 Jan])';
}

Configurable Flags

Adjust script behavior during testing or development:

const DEV_CONFIG = {
    SEND_ERROR_MAIL: true,         // Send email for errors/warnings
    CREATE_FAILURE_RECORD: true,   // Create failure record in sheet

    // Processing filters:
    // 1. Email must be unread
    // 2. Must have an INCLUDE label and no EXCLUDE label
    // 3. Must not already exist in the sheet
    RERUN_READ_MAILS: false,       // Bypass filter 1: reprocess read emails
    RERUN_PROCESSED_IN_SHEET: true,// Bypass filter 3: reprocess even if in sheet
    RERUN_PROCESSED_IN_EMAIL: false,// Alter filter 2: include PROCESSED emails
    // WARNING: Setting RERUN_PROCESSED_IN_EMAIL=true picks up entire history

    MARK_AS_PROCESSED: true,       // Mark as read + apply PROCESSED label
    SANITY_TESTS_RUN: false,       // Only run TESTCASES label
    OUTPUT_SHEET_TITLE: CONFIG.MAIN_SHEET_NAME,
}

Code Flow

  1. Fetches email threads based on labels or search queries
  2. Fetches existing sheet rows once (O(n) duplicate detection)
  3. For each unread email in each thread:
    • Populates email data (subject, body, date, SMS detection)
    • For HTML-whitelisted rules, parses HTML body instead of plain text
    • Identifies the applicable rule via subject mapping or SMS body matching
    • Extracts transaction fields using the rule's regex map
    • Resolves account names, categories, subcategories, and budget
    • For transfers: builds a separate transfer payload
    • Normalizes the transaction title
    • Validates mandatory fields
    • Checks for duplicates against existing + newly-added rows
    • Appends row to Google Sheet and updates in-memory array
  4. Builds and sends success email with:
    • Normal transactions section (batch approve/edit URLs)
    • Transfer transactions section (per-row approve links)
    • Processing summary
  5. Sends error emails for any failures
  6. Labels processed emails

Script File Order

Files in Google Apps Script are evaluated in order. Maintain this sequence:

  1. 1-constants.gs -- Constants, enums, config objects
  2. 2-userConfig.gs -- User-specific regex rules, categories, accounts, secrets
  3. 3-main.gs -- Main entry point and processing loop
  4. 4-utils.gs -- All utility functions (parsing, email, sheets)
  5. 5-test.gs -- Dev config, test helpers, sanity test framework

Alternative: Use the appsscript.json manifest to configure execution order explicitly.

Account Mapping

Map account identifiers to display names and enrich with secrets:

const baseAccountIdsMap = {
    "BANKA_SAVINGS": ["BankA"],
    "BANKA_CC": ["BankA credit card"],
};

const accountIdsToDisplayNameMap = {
    "BANKA_SAVINGS": "BankA",
    "BANKA_CC": "BankA CC",
};

// Enriches with secret identifiers stored in Script Properties
const accountKeywordMap = deepFreeze(
    enrichMapWithSecrets(baseAccountIdsMap, accountIdsToDisplayNameMap)
);

The enrichMapWithSecrets function:

  • Constructs secret keys as ACCOUNT_ID_<KEY> (e.g., ACCOUNT_ID_BANKA_SAVINGS)
  • Merges secret values with base keywords
  • Replaces base keys with display names from accountIdsToDisplayNameMap
  • Deduplicates all arrays

Gmail Labeling Strategy

TPS applies labels to individual emails (not threads) using the Gmail API. This is a deliberate choice over the simpler thread-level labeling, and it matters when things go wrong.

Why per-email instead of per-thread?

A single Gmail thread can contain multiple emails (e.g., a bank sends transaction alerts as replies in the same thread). If one email parses successfully but another fails, per-email labeling means:

  • The successful email gets Txs/✅ (processed) and is marked as read
  • The failed email gets Txs/🛠️ (to fix) and stays unread
  • On the next run, TPS picks up the thread again (because it still has unprocessed emails), iterates through all emails in it, skips the read/processed ones, and only retries the unread ones

If labels were applied at the thread level, the entire thread would be marked as processed and the failed email would be silently skipped forever.

Conversation view tip: Go to Gmail > Settings > General > Conversation view and select "Conversation view off". This lets you see each email individually in your inbox, making it easier to spot which specific emails have which labels.

See the Gmail API docs for setup.