Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/backend-governance.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ jobs:
- name: Install dependencies
run: npm ci

- name: Validate Prisma schema and migration consistency
run: npm run prisma:schema-check

- name: Apply PostgreSQL migrations
run: npm run db:migrate

Expand Down
17 changes: 17 additions & 0 deletions backend/LATENCY_MONITORING.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ The latency monitoring system tracks P95 latency for all API endpoints and sends
- **SLO breach detection** with automatic alerting
- **Multiple alert integrations** (Slack, PagerDuty)
- **Configurable thresholds** via environment variables
- **Prometheus request error metrics** for error-budget burn alerts
- **External dependency latency and failure metrics** from health probes
- **Endpoint normalization** for dynamic routes
- **Admin endpoints** for monitoring status

Expand Down Expand Up @@ -100,6 +102,21 @@ Service: YieldVault Backend

## Monitoring Endpoints

Prometheus alert rules are versioned in
`backend/monitoring/prometheus-alerts.yml`. Load this file from the Prometheus
configuration used by the deployment. It alerts when critical endpoint P95
latency exceeds the endpoint registry budget, when 5xx traffic burns more than
1% of the critical endpoint error budget over five minutes, or when an external
dependency probe has a P95 latency above two seconds.

The `/metrics` endpoint exposes:

- `http_request_error_total`, labeled by method, normalized route, status code,
and status class.
- `external_dependency_latency_seconds`, labeled by dependency, operation, and
success or failure outcome.
- `external_dependency_error_total`, labeled by dependency and operation.

### Admin Endpoints (API Key Required)

#### GET `/admin/latency-status`
Expand Down
37 changes: 37 additions & 0 deletions backend/docs/PRISMA_MIGRATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Run from `backend/`:
| Script | Purpose |
| --- | --- |
| `npm run prisma:generate` | Regenerate the Prisma client |
| `npm run prisma:schema-check` | Validate the schema and fail if committed migrations do not produce it |
| `npm run prisma:migrate -- --name add_foo` | Create a migration from `schema.prisma` (dev) |
| `npm run prisma:deploy` | Apply pending migrations (CI / prod) |
| `npm run prisma:status` | Show applied vs pending |
Expand All @@ -41,6 +42,15 @@ Create a change:
change is reversible.
5. Commit schema, migration, and rollback together.

The backend governance workflow runs `prisma:schema-check` on every pull
request that changes the backend or migration workflow. The check validates
`schema.prisma`, then compares the final state represented by
`prisma/migrations` with the schema datamodel. A non-zero result means the
schema was changed without a matching migration, a migration was edited after
being applied, or a migration folder is incomplete. Do not bypass this check
by editing generated or deployed database state; create and commit the proper
forward migration instead.

Existing schema is already captured by `0_init` plus follow-up migrations.
New environments run `prisma:deploy` (SQLite/dev) or `db:migrate` (Postgres).

Expand Down Expand Up @@ -73,3 +83,30 @@ Prisma does not generate automatic down migrations. Rollback is explicit:

Never `DROP` or `TRUNCATE` in a canary window. Follow
`docs/CANARY_MIGRATION_STRATEGY.md`.

## Drift recovery

When CI reports schema drift:

1. Run `npm run prisma:schema-check` from `backend/` and inspect the diff.
2. If `schema.prisma` is ahead, create a named migration with
`npm run prisma:migrate -- --name <description>` and commit its
`migration.sql`.
3. If a committed migration was edited after deployment, restore its original
SQL and create a new corrective migration. Never rewrite an applied
migration to make the check pass.
4. Check the target with `npm run prisma:status`. For a failed deployment,
resolve the migration only after confirming whether any SQL was applied.
5. Use the documented rollback SQL or restore a verified backup for destructive
changes, then deploy the corrective forward migration and rerun the checks.

For an unapplied migration, use:

```bash
npx prisma migrate resolve --schema prisma/schema.prisma --rolled-back <name>
```

For an applied migration with a rollback file, use
`npm run prisma:rollback -- --name <name>` and follow the script output. Take a
backup and coordinate production recovery before running destructive rollback
SQL.
67 changes: 67 additions & 0 deletions backend/monitoring/prometheus-alerts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
groups:
- name: yieldvault-backend-slos
interval: 30s
rules:
- alert: YieldVaultHealthLatencyHigh
expr: histogram_quantile(0.95, sum by (route, method, le) (rate(http_response_time_seconds_bucket{route="/health"}[5m]))) > 0.05
for: 5m
labels:
severity: warning
service: yieldvault-backend
annotations:
summary: "YieldVault health endpoint latency budget exceeded"
description: "The /health P95 latency is above its 50ms budget."

- alert: YieldVaultReadinessLatencyHigh
expr: histogram_quantile(0.95, sum by (route, method, le) (rate(http_response_time_seconds_bucket{route="/ready"}[5m]))) > 0.1
for: 5m
labels:
severity: warning
service: yieldvault-backend
annotations:
summary: "YieldVault readiness latency budget exceeded"
description: "The /ready P95 latency is above its 100ms budget."

- alert: YieldVaultCriticalReadLatencyHigh
expr: histogram_quantile(0.95, sum by (route, method, le) (rate(http_response_time_seconds_bucket{route="/api/v1/vault/summary"}[5m]))) > 0.2
for: 5m
labels:
severity: warning
service: yieldvault-backend
annotations:
summary: "YieldVault summary latency budget exceeded"
description: "The vault summary P95 latency is above its 200ms budget."

- alert: YieldVaultCriticalWriteLatencyHigh
expr: histogram_quantile(0.95, sum by (route, method, le) (rate(http_response_time_seconds_bucket{route=~"/api/v1/vault/(deposit|withdraw)"}[5m]))) > 0.5
for: 5m
labels:
severity: warning
service: yieldvault-backend
annotations:
summary: "YieldVault write latency budget exceeded"
description: "A critical vault write endpoint has P95 latency above its 500ms budget."

- alert: YieldVaultCriticalEndpointErrorBudgetBurn
expr: |
sum by (route) (rate(http_request_error_total{status_class="5xx",route=~"/health|/ready|/api/v1/vault/(summary|deposit|withdraw)"}[5m]))
/
sum by (route) (rate(http_request_count{route=~"/health|/ready|/api/v1/vault/(summary|deposit|withdraw)"}[5m]))
> 0.01
for: 5m
labels:
severity: critical
service: yieldvault-backend
annotations:
summary: "YieldVault critical endpoint error budget is burning"
description: "5xx responses exceed 1% of traffic for {{ $labels.route }} over the last 5 minutes."

- alert: YieldVaultExternalDependencyLatencyHigh
expr: histogram_quantile(0.95, sum by (dependency, operation, le) (rate(external_dependency_latency_seconds_bucket{outcome="success"}[5m]))) > 2
for: 5m
labels:
severity: warning
service: yieldvault-backend
annotations:
summary: "YieldVault external dependency is slow"
description: "P95 {{ $labels.dependency }} {{ $labels.operation }} latency is above 2 seconds."
1 change: 1 addition & 0 deletions backend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
"db:check-drift": "node scripts/check-postgres-drift.js",
"adrs:check": "node scripts/check-adrs.js",
"prisma:generate": "prisma generate --schema prisma/schema.prisma",
"prisma:schema-check": "prisma validate --schema prisma/schema.prisma && prisma migrate diff --from-migrations prisma/migrations --to-schema-datamodel prisma/schema.prisma --exit-code",
"prisma:migrate": "prisma migrate dev --schema prisma/schema.prisma",
"prisma:deploy": "prisma migrate deploy --schema prisma/schema.prisma",
"prisma:status": "prisma migrate status --schema prisma/schema.prisma",
Expand Down
3 changes: 3 additions & 0 deletions backend/src/healthProbe.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { logger } from './middleware/structuredLogging';
import { observeExternalDependency } from './metrics';

export interface DependencyProbeState {
status: 'up' | 'down' | 'degraded';
Expand Down Expand Up @@ -52,6 +53,7 @@ class HealthProbeService {
try {
const result = await registration.probe();
const latencyMs = Date.now() - startMs;
observeExternalDependency(name, 'health_probe', latencyMs, result === 'up' ? 'success' : 'failure');

state.status = result;
state.latencyMs = latencyMs;
Expand All @@ -66,6 +68,7 @@ class HealthProbeService {
}
} catch (error) {
const latencyMs = Date.now() - startMs;
observeExternalDependency(name, 'health_probe', latencyMs, 'failure');
state.status = 'down';
state.latencyMs = latencyMs;
state.lastCheckedAt = new Date().toISOString();
Expand Down
7 changes: 7 additions & 0 deletions backend/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ import {
register,
httpRequestCount,
httpResponseTime,
httpRequestErrorCount,
activeConnections,
updateVaultMetrics,
syncJobGovernanceMetrics,
Expand Down Expand Up @@ -674,6 +675,12 @@ app.use((req: Request, res: Response, next: NextFunction) => {

httpRequestCount.inc(labels);
httpResponseTime.observe(labels, durationSeconds);
if (res.statusCode >= 400) {
httpRequestErrorCount.inc({
...labels,
status_class: `${Math.floor(res.statusCode / 100)}xx`,
});
}
if (res.statusCode === 429) {
rateLimitEvents.inc({ tier: req.apiVersion ?? 'global', outcome: 'limited' });
}
Expand Down
37 changes: 37 additions & 0 deletions backend/src/metrics.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,43 @@ export const httpResponseTime = new Histogram({
registers: [register],
});

export const httpRequestErrorCount = new Counter({
name: 'http_request_error_total',
help: 'Total HTTP requests completed with a client or server error status',
labelNames: ['method', 'route', 'status_code', 'status_class'],
registers: [register],
});

export const externalDependencyLatency = new Histogram({
name: 'external_dependency_latency_seconds',
help: 'Latency of external dependency probes and calls in seconds',
labelNames: ['dependency', 'operation', 'outcome'],
buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30],
registers: [register],
});

export const externalDependencyErrorCount = new Counter({
name: 'external_dependency_error_total',
help: 'Total failed external dependency probes and calls',
labelNames: ['dependency', 'operation'],
registers: [register],
});

export function observeExternalDependency(
dependency: string,
operation: string,
durationMs: number,
outcome: 'success' | 'failure',
): void {
externalDependencyLatency.observe(
{ dependency, operation, outcome },
durationMs / 1000,
);
if (outcome === 'failure') {
externalDependencyErrorCount.inc({ dependency, operation });
}
}

export const activeConnections = new Gauge({
name: 'http_active_connections',
help: 'Number of active HTTP connections',
Expand Down
41 changes: 29 additions & 12 deletions frontend/src/components/VaultDashboard.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,20 @@ import { useDelayedLoading } from "../hooks/useDelayedLoading";
import { useVault } from "../context/VaultContext";
import ApiStatusBanner from "./ApiStatusBanner";
import SharePriceDisplay from "./SharePriceDisplay";
import VaultPositionBalance from "./VaultPositionBalance";
import VaultPerformanceChart from "./VaultPerformanceChart";
import { useToast } from "../context/ToastContext";
import { Tabs, TabsContent, TabsList, TabsTrigger } from "./Tabs";
import { FormField } from "../forms";
import { isApiError, isValidationError } from "../lib/api";
import { isValidationError } from "../lib/api";
import { useForm } from "../forms/useForm";
import { validate, type ValidationSchema } from "../forms/validate";
import { useDepositMutation, useWithdrawMutation } from "../hooks/useVaultMutations";
import { useTokenAllowance } from "../hooks/useTokenAllowance";
import { usePortfolioHoldings } from "../hooks/usePortfolioData";
import { createDepositFormSchema, MIN_DEPOSIT_AMOUNT, MAX_DEPOSIT_AMOUNT, USDC_DISPLAY_DECIMALS } from "../forms/schemas/depositFormSchema";
import { createWithdrawFormSchema } from "../forms/schemas/withdrawFormSchema";
import { mapServerError } from "../lib/errorMappers";
import { mapServerError, mapTransactionError } from "../lib/errorMappers";
import confetti from "canvas-confetti";
import CopyButton from "./CopyButton";
import { Button } from "./ui/Button";
Expand Down Expand Up @@ -226,7 +227,7 @@ const VaultDashboard: React.FC<VaultDashboardProps> = ({
});
const { isStale: statsIsStale, ageText: statsAgeText } = useStaleIndicator(lastUpdate);

const { data: portfolioHoldings } = usePortfolioHoldings(walletAddress);
const { data: portfolioHoldings, isLoading: isPortfolioLoading } = usePortfolioHoldings(walletAddress);

// Deposit balance comes from the connected wallet's USDC balance; withdraw
// balance is the user's current vault position (sum of holdings value).
Expand All @@ -244,6 +245,8 @@ const VaultDashboard: React.FC<VaultDashboardProps> = ({
txHash?: string;
retryable?: boolean;
actionType?: TransactionTab;
technicalCode?: string;
supportReference?: string;
} | null>(null);
const [retryCount, setRetryCount] = useState(0);
const [mobileActionsOpen, setMobileActionsOpen] = useState(false);
Expand Down Expand Up @@ -704,20 +707,20 @@ const VaultDashboard: React.FC<VaultDashboardProps> = ({
dashboardUrl.setStep("amount");
}

let errorMessage = t("vaultDashboard.toast.genericError");
const transactionError = mapTransactionError(err);
const errorMessage = t(`vaultDashboard.toast.transactionErrors.${transactionError.kind}`);

if (isValidationError(err)) {
errorMessage = err.details?.[0]?.message || errorMessage;
} else if (err instanceof Error) {
errorMessage = err.message;
} else if (mappedError.generalError) {
errorMessage = mappedError.generalError;
}
console.error("Vault transaction failed", {
action: actionType,
error: err,
technicalCode: transactionError.technicalCode,
supportReference: transactionError.supportReference,
});

// Field-level validation failures need corrected input, not a blind resubmit.
// Everything else (network hiccups, RPC timeouts, transient 5xx) is worth retrying.
const retryable =
!hasFieldErrors && !isValidationError(err) && (isApiError(err) ? err.retryable : true);
!hasFieldErrors && !isValidationError(err) && transactionError.retryable;

if (options.isRetry) {
setRetryCount((count) => count + 1);
Expand All @@ -728,6 +731,8 @@ const VaultDashboard: React.FC<VaultDashboardProps> = ({
message: errorMessage,
retryable,
actionType,
technicalCode: transactionError.technicalCode,
supportReference: transactionError.supportReference,
});

toast.error({
Expand Down Expand Up @@ -865,6 +870,13 @@ const VaultDashboard: React.FC<VaultDashboardProps> = ({
}}
/>

{walletAddress && (
<VaultPositionBalance
holding={(portfolioHoldings ?? []).find((holding) => holding.symbol === "yvUSDC")}
isLoading={isPortfolioLoading}
/>
)}

{/* Per-widget refresh control + stale indicator for stats panel */}
<div style={{ marginBottom: "16px" }}>
<RefreshControl
Expand Down Expand Up @@ -1785,6 +1797,11 @@ const VaultDashboard: React.FC<VaultDashboardProps> = ({
<p style={{ color: "var(--text-secondary)", marginBottom: "8px", maxWidth: "300px" }}>
{transactionResult?.message}
</p>
{transactionResult?.supportReference && (
<p style={{ color: "var(--text-tertiary)", marginBottom: "8px", fontSize: "0.75rem" }}>
Reference: {transactionResult.supportReference}
</p>
)}

{!transactionResult?.success && retryCount >= MAX_TRANSACTION_RETRY_ATTEMPTS && (
<p
Expand Down
Loading
Loading