Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions backend/src/config/supabase.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
// Re-export supabase client from database.js for backward compatibility
export { supabase } from './database.js';
36 changes: 36 additions & 0 deletions backend/src/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,42 @@ app.get("/health", (req, res) => {
});
});

// Version endpoint
app.get("/version", (req, res) => {
res.json({
version: "1.0.0",
name: "ai-grant-crawler-backend",
buildTime: process.env.BUILD_TIME || new Date().toISOString(),
nodeVersion: process.version,
environment: env.nodeEnv,
});
Comment on lines +60 to +66
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n backend/src/server.js | head -80

Repository: Datakult0r/ai-grant-crawler-a2a-pro

Length of output: 3182


🏁 Script executed:

rg "BUILD_TIME" --type js --type json --type yaml --type sh

Repository: Datakult0r/ai-grant-crawler-a2a-pro

Length of output: 167


🏁 Script executed:

cat -n backend/src/config/env.js | head -50

Repository: Datakult0r/ai-grant-crawler-a2a-pro

Length of output: 1628


🏁 Script executed:

cat -n backend/src/config/env.js

Repository: Datakult0r/ai-grant-crawler-a2a-pro

Length of output: 2556


Make buildTime stable across requests.

The buildTime value is computed on every request since BUILD_TIME is not set as an environment variable. This means the endpoint returns the current time instead of the actual build time, and recalculates it unnecessarily for each request. Compute it once at module startup.

🔧 Suggested fix
-// Version endpoint
-app.get("/version", (req, res) => {
+const buildTime = process.env.BUILD_TIME || new Date().toISOString();
+// Version endpoint
+app.get("/version", (req, res) => {
   res.json({
     version: "1.0.0",
     name: "ai-grant-crawler-backend",
-    buildTime: process.env.BUILD_TIME || new Date().toISOString(),
+    buildTime,
     nodeVersion: process.version,
     environment: env.nodeEnv,
   });
 });
🤖 Prompt for AI Agents
In `@backend/src/server.js` around lines 60 - 66, The buildTime is being
recomputed on every request; move its computation to module startup by creating
a module-level constant (e.g., BUILD_TIME or buildTimeAtStartup) set to
process.env.BUILD_TIME || new Date().toISOString() and then use that constant in
the res.json call (where buildTime is currently computed inline). Update the
response in server.js to reference the new constant (instead of calling new
Date().toISOString() per request) so the value is stable across requests.

});

// Metrics endpoint (basic)
const startTime = Date.now();
let requestCount = 0;
app.use((req, res, next) => {
requestCount++;
next();
});
Comment on lines +69 to +75
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if the file exists and get its size
if [ -f "backend/src/server.js" ]; then
  echo "=== File found ==="
  wc -l backend/src/server.js
else
  echo "File not found at backend/src/server.js"
  # Search for server.js files
  find . -name "server.js" -type f 2>/dev/null
fi

Repository: Datakult0r/ai-grant-crawler-a2a-pro

Length of output: 123


🏁 Script executed:

#!/bin/bash
# Read the entire server.js file to see the code structure and route order
cat -n backend/src/server.js

Repository: Datakult0r/ai-grant-crawler-a2a-pro

Length of output: 5254


Move the request counter middleware before the /health and /version routes to count all requests.

The middleware at line 72 is registered after these routes, so requests to /health and /version never increment requestCount. Move the middleware to lines 48-51 (after rate limiting, before route definitions) if you intended to count all requests.

🤖 Prompt for AI Agents
In `@backend/src/server.js` around lines 69 - 75, The request-counting middleware
(the module that increments requestCount using app.use) is registered after the
/health and /version route handlers so those endpoints are not counted; move the
requestCount and its app.use middleware (the block that declares startTime and
let requestCount = 0 and the app.use((req, res, next) => { requestCount++;
next(); })) to earlier in the middleware chain—specifically immediately after
rate limiting and before any route definitions (i.e., before the /health and
/version route registrations) so all incoming requests are counted.


app.get("/metrics", (req, res) => {
const uptime = Math.floor((Date.now() - startTime) / 1000);
const memoryUsage = process.memoryUsage();

res.json({
uptime: uptime,
uptimeFormatted: `${Math.floor(uptime / 3600)}h ${Math.floor((uptime % 3600) / 60)}m ${uptime % 60}s`,
requestCount: requestCount,
memory: {
heapUsed: Math.round(memoryUsage.heapUsed / 1024 / 1024) + " MB",
heapTotal: Math.round(memoryUsage.heapTotal / 1024 / 1024) + " MB",
rss: Math.round(memoryUsage.rss / 1024 / 1024) + " MB",
},
environment: env.nodeEnv,
});
});

// API routes
app.use("/api/grants", grantsRouter);
app.use("/api/crawler", crawlerRouter);
Expand Down
16 changes: 16 additions & 0 deletions backend/src/services/emailService.js
Original file line number Diff line number Diff line change
Expand Up @@ -229,3 +229,19 @@ export async function sendEmail(to, subject, html) {
return { success: false, error: err.message };
}
}

// Wrapper functions for notification scheduler
export async function notifyDeadlineApproaching(email, grant, daysRemaining, unsubscribeToken = 'default') {
const { subject, html } = createDeadlineAlertEmail(grant, daysRemaining, unsubscribeToken);
return sendEmail(email, subject, html);
}

export async function notifyNewHighRelevanceGrant(email, grant, unsubscribeToken = 'default') {
const { subject, html } = createNewGrantAlertEmail([grant], unsubscribeToken);
return sendEmail(email, subject, html);
}

export async function sendWeeklyDigest(email, grants, proposals, stats, unsubscribeToken = 'default') {
const { subject, html } = createWeeklyDigestEmail(stats, grants.slice(0, 5), unsubscribeToken);
return sendEmail(email, subject, html);
}
Comment on lines +244 to +247
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Unused proposals parameter and missing null guard on grants.

  1. The proposals parameter is declared but never used. Either remove it or document it as reserved for future use.
  2. grants.slice(0, 5) will throw a TypeError if grants is null or undefined.
🛠️ Proposed fix
-export async function sendWeeklyDigest(email, grants, proposals, stats, unsubscribeToken = 'default') {
-  const { subject, html } = createWeeklyDigestEmail(stats, grants.slice(0, 5), unsubscribeToken);
+export async function sendWeeklyDigest(email, grants, stats, unsubscribeToken = 'default') {
+  const topGrants = Array.isArray(grants) ? grants.slice(0, 5) : [];
+  const { subject, html } = createWeeklyDigestEmail(stats, topGrants, unsubscribeToken);
   return sendEmail(email, subject, html);
 }

If proposals is intentionally reserved for future use, keep it but add a JSDoc comment explaining its purpose.

🤖 Prompt for AI Agents
In `@backend/src/services/emailService.js` around lines 244 - 247, In
sendWeeklyDigest replace the unused proposals parameter or document it as
reserved (add a JSDoc comment on sendWeeklyDigest if you want to keep it for
future use) and guard against null/undefined grants before slicing: ensure
grants is an array (e.g., default to [] or validate with Array.isArray) before
calling grants.slice(0, 5); call createWeeklyDigestEmail with the safe grants
array so createWeeklyDigestEmail and sendEmail usage remain unchanged.

24 changes: 22 additions & 2 deletions backend/src/services/firecrawl.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,30 @@ dotenv.config();

const apiKey = process.env.FIRECRAWL_API_KEY;
if (!apiKey) {
console.warn("⚠️ FIRECRAWL_API_KEY missing in .env");
console.warn("⚠️ FIRECRAWL_API_KEY missing in .env - Firecrawl features will be disabled");
}

const firecrawl = new FirecrawlApp({ apiKey: apiKey });
// Only initialize Firecrawl if API key is available
const firecrawl = apiKey ? new FirecrawlApp({ apiKey: apiKey }) : null;

export class FirecrawlService {
/**
* Check if Firecrawl is available
*/
static isAvailable() {
return firecrawl !== null;
}

/**
* Scrape a single grant page to get its content.
* @param {string} url
* @returns {Promise<Object>} Formatted markdown and metadata
*/
static async scrapeGrantPage(url) {
if (!firecrawl) {
console.warn('[FIRECRAWL] Service not available - API key missing');
return { success: false, error: 'Firecrawl API key not configured' };
}
try {
const scrapeResult = await firecrawl.scrapeUrl(url, {
formats: ["markdown"],
Expand All @@ -34,6 +46,10 @@ export class FirecrawlService {
* @returns {Promise<Array>} List of discovered URLs
*/
static async crawlGrantSite(url, limit = 10) {
if (!firecrawl) {
console.warn('[FIRECRAWL] Service not available - API key missing');
return { success: false, error: 'Firecrawl API key not configured' };
}
try {
const crawlResponse = await firecrawl.crawlUrl(url, {
limit: limit,
Expand Down Expand Up @@ -62,6 +78,10 @@ export class FirecrawlService {
* @returns {Promise<Object>} Search results
*/
static async searchGrantContext(query) {
if (!firecrawl) {
console.warn('[FIRECRAWL] Service not available - API key missing');
return { success: false, error: 'Firecrawl API key not configured' };
}
try {
const params = {
pageOptions: {
Expand Down
66 changes: 59 additions & 7 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,11 @@ This guide covers deploying the AI Grant Crawler application to production.
The application consists of two main components:

1. **Frontend** (SvelteKit) - Deployed to Vercel (automatic via GitHub integration)
2. **Backend** (Node.js/Express) - Deployed to Fly.io (manual setup required)
2. **Backend** (Node.js/Express) - Deployed to Railway or Fly.io

## Prerequisites

- [Fly.io CLI](https://fly.io/docs/hands-on/install-flyctl/) installed
- Fly.io account created
- Railway account OR [Fly.io CLI](https://fly.io/docs/hands-on/install-flyctl/) installed
- Supabase project with database schema applied
- API keys for Gemini (required) and OpenRouter (optional)

Expand All @@ -28,7 +27,46 @@ Set these in your Vercel project settings:
PUBLIC_API_URL=https://your-backend.fly.dev/api
```

## Backend Deployment (Fly.io)
## Backend Deployment (Railway) - Recommended

Railway is the recommended deployment platform for the backend. A `railway.json` configuration file is already included.

### 1. Connect Repository

1. Go to [Railway Dashboard](https://railway.app/dashboard)
2. Click "New Project" > "Deploy from GitHub repo"
3. Select `Datakult0r/ai-grant-crawler-a2a-pro`
4. Choose the `backend` directory as the root

### 2. Set Environment Variables

In Railway Dashboard > Project > Variables:

```
# Required
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_ANON_KEY=your-anon-key
GEMINI_API_KEY=your-gemini-key
PORT=3000

# Optional
OPENROUTER_API_KEY=your-openrouter-key
LOW_COST_MODE=true
AI_RESEARCHER_ENABLED=false
IS_DEMO=false
```

### 3. Deploy

Railway will automatically deploy when you push to the connected branch. You can also trigger manual deploys from the dashboard.

### 4. Get Your Backend URL

After deployment, Railway provides a URL like `https://your-app.up.railway.app`. Use this for the frontend's `PUBLIC_API_URL`.

---

## Backend Deployment (Fly.io) - Alternative

### 1. Install Fly CLI

Expand Down Expand Up @@ -130,9 +168,23 @@ Access your app's metrics and logs at: https://fly.io/apps/your-app-name

### Health Check

The backend exposes a `/health` endpoint that returns:
- Database connectivity status
- API key configuration status
The backend exposes several monitoring endpoints:

**`/health`** - Basic health check
- Returns: `{ status: "ok", timestamp, environment }`

**`/version`** - Application version info
- Returns: `{ version, name, buildTime, nodeVersion, environment }`

**`/metrics`** - Runtime metrics
- Returns: `{ uptime, uptimeFormatted, requestCount, memory: { heapUsed, heapTotal, rss }, environment }`

Example:
```bash
curl https://your-app.fly.dev/health
curl https://your-app.fly.dev/version
curl https://your-app.fly.dev/metrics
```
Comment on lines +171 to +187
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Clarify /metrics memory units/format in docs.

The implementation returns memory values as strings with “ MB”, but the doc implies raw values. Please document the units/format (or adjust the API to return numeric values with explicit units).

📝 Suggested doc tweak
 **`/metrics`** - Runtime metrics
-- Returns: `{ uptime, uptimeFormatted, requestCount, memory: { heapUsed, heapTotal, rss }, environment }`
+- Returns: `{ uptime, uptimeFormatted, requestCount, memory: { heapUsed, heapTotal, rss }, environment }` (memory values are strings in MB)
🤖 Prompt for AI Agents
In `@docs/DEPLOYMENT.md` around lines 133 - 149, Update the /metrics documentation
to explicitly state that memory fields (memory.heapUsed, memory.heapTotal,
memory.rss) are returned as formatted strings in megabytes with an " MB" suffix
(e.g., "123.45 MB") and that uptimeFormatted is a human-readable string, or
alternatively change the API to return numeric byte values plus an explicit unit
field; specifically, either adjust the docs under the `/metrics` section to show
the memory fields as strings with " MB" units and an example response like
memory: { heapUsed: "123.45 MB", heapTotal: "256.00 MB", rss: "300.12 MB" } or
modify the /metrics response implementation to return numeric values (bytes) and
add a memory.unit = "bytes" field so the contract is unambiguous.


## Troubleshooting

Expand Down
Loading